CodeSource
/
Readur
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Readur/docs/architecture.md

843 lines
23 KiB
Markdown
Raw Blame History

# Architecture Overview
This document provides a comprehensive overview of Readur's system architecture, component interactions, data flows, and design decisions.
## System Components
### High-Level Architecture
**Important:** Readur is designed as a single-instance, monolithic application. It does NOT support multiple server instances, clustering, or high availability configurations.
```
┌──────────────────────────────────────────────────────────────────┐
│ Readur Single Instance │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌──────────────────────┐ │
│ │ Web Server │ │ Business │ │ Background Services │ │
│ │ (Axum) │ │ Logic │ │ - OCR Worker │ │
│ └─────────────┘ └─────────────┘ │ - File Watcher │ │
│ │ - Queue Processor │ │
│ └──────────────────────┘ │
└───────────────────────────┬──────────────────────────────────────┘
│
┌───────────────────▼─────────────────────┐
│ Data Layer │
│ ┌────────┐ ┌────────┐ ┌──────────┐ │
│ │Database│ │Storage │ │ Queue │ │
│ │ (PG) │ │(S3/FS) │ │(DB-based) │ │
│ └────────┘ └────────┘ └──────────┘ │
└──────────────────────────────────────────┘
```
### Component Breakdown
```
Readur Application Instance
├── Web Server (Axum)
│ ├── HTTP API Endpoints
│ ├── WebSocket Server
│ ├── Static File Server
│ └── Middleware Stack
├── Business Logic Layer
│ ├── Document Management
│ ├── Search Engine
│ ├── User Management
│ ├── OCR Processing
│ └── Source Synchronization
├── Data Access Layer
│ ├── Database Connection Pool
│ ├── File Storage Interface
│ ├── Cache Layer
│ └── External API Clients
└── Background Services
├── OCR Queue Worker
├── File Watcher
├── Source Scheduler
└── Cleanup Tasks
```
## Data Flow Architecture
### Document Upload Flow
```
User Upload Request
│
▼
[1] Nginx/Reverse Proxy
│
├─── Rate Limiting
├─── Request Validation
└─── Load Balancing
│
▼
[2] Authentication Middleware
│
├─── JWT Validation
└─── Permission Check
│
▼
[3] File Upload Handler
│
├─── File Type Validation
├─── Size Validation
└─── Virus Scanning (optional)
│
▼
[4] Storage Service
│
├─── Generate UUID
├─── Calculate Hash
└─── Store File
│
▼
[5] Database Transaction
│
├─── Create Document Record
├─── Add Metadata
└─── Queue for OCR
│
▼
[6] OCR Queue
│
├─── Priority Assignment
└─── Worker Notification
│
▼
[7] Response to Client
│
└─── Document ID + Status
```
### OCR Processing Pipeline
```
OCR Queue Entry
│
▼
[1] Queue Worker Pickup
│
├─── Lock Document
└─── Update Status
│
▼
[2] File Retrieval
│
├─── Load from Storage
└─── Verify Integrity
│
▼
[3] Preprocessing
│
├─── Image Enhancement
├─── Format Conversion
└─── Page Splitting
│
▼
[4] OCR Engine (Tesseract)
│
├─── Language Detection
├─── Text Extraction
└─── Confidence Scoring
│
▼
[5] Post-processing
│
├─── Text Cleaning
├─── Format Normalization
└─── Metadata Extraction
│
▼
[6] Database Update
│
├─── Store Extracted Text
├─── Update Search Index
└─── Record Metrics
│
▼
[7] Notification
│
├─── WebSocket Update
└─── Email (if configured)
```
### Search Request Flow
```
Search Query
│
▼
[1] Query Parser
│
├─── Tokenization
├─── Stemming
└─── Query Expansion
│
▼
[2] Search Executor
│
├─── Full-Text Search (PostgreSQL)
├─── Filter Application
└─── Ranking Algorithm
│
▼
[3] Result Processing
│
├─── Snippet Generation
├─── Highlighting
└─── Facet Calculation
│
▼
[4] Permission Filter
│
└─── User Access Check
│
▼
[5] Response Assembly
│
├─── Pagination
├─── Metadata Enrichment
└─── JSON Serialization
```
## Queue Architecture
### OCR Queue System
```sql
-- Queue table structure
CREATE TABLE ocr_queue (
id UUID PRIMARY KEY,
document_id UUID REFERENCES documents(id),
status VARCHAR(20), -- pending, processing, completed, failed
priority INTEGER DEFAULT 5,
retry_count INTEGER DEFAULT 0,
max_retries INTEGER DEFAULT 3,
created_at TIMESTAMP,
started_at TIMESTAMP,
completed_at TIMESTAMP,
error_message TEXT,
worker_id VARCHAR(100)
);
-- Efficient queue fetching with SKIP LOCKED
SELECT * FROM ocr_queue
WHERE status = 'pending'
ORDER BY priority DESC, created_at ASC
FOR UPDATE SKIP LOCKED
LIMIT 1;
```
### Queue Worker Architecture
```rust
// Queue processing with fixed thread pools
pub struct OcrQueueService {
pool: PgPool,
workers: Vec<JoinHandle<()>>,
shutdown: Arc<AtomicBool>,
}
impl OcrQueueService {
pub async fn start_workers(&self) {
// Fixed thread allocation:
// - OCR runtime: 3 threads
// - Background runtime: 2 threads
// - Database runtime: 2 threads
let ocr_workers = 3;
for worker_id in 0..ocr_workers {
let pool = self.pool.clone();
let shutdown = self.shutdown.clone();
let handle = tokio::spawn(async move {
while !shutdown.load(Ordering::Relaxed) {
if let Some(job) = fetch_next_job(&pool).await {
process_ocr_job(job, &pool).await;
} else {
tokio::time::sleep(Duration::from_secs(1)).await;
}
}
});
self.workers.push(handle);
}
}
}
```
## Storage Architecture
### Storage Abstraction Layer
```rust
// Storage trait for multiple backends
#[async_trait]
pub trait StorageBackend: Send + Sync {
async fn store(&self, key: &str, data: &[u8]) -> Result<()>;
async fn retrieve(&self, key: &str) -> Result<Vec<u8>>;
async fn delete(&self, key: &str) -> Result<()>;
async fn exists(&self, key: &str) -> Result<bool>;
async fn list(&self, prefix: &str) -> Result<Vec<String>>;
}
// Implementations
pub struct LocalStorage { base_path: PathBuf }
pub struct S3Storage { bucket: String, client: S3Client }
pub struct AzureStorage { container: String, client: BlobClient }
```
### File Organization
```
Storage Root/
├── documents/
│ ├── {year}/{month}/{day}/
│ │ └── {uuid}.{extension}
├── thumbnails/
│ ├── {year}/{month}/{day}/
│ │ └── {uuid}_thumb.jpg
├── processed/
│ ├── ocr/
│ │ └── {uuid}_ocr.txt
│ └── metadata/
│ └── {uuid}_meta.json
└── temp/
└── {session_id}/
└── {temp_files}
```
## Database Schema
### Core Tables
```sql
-- Users table
CREATE TABLE users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
username VARCHAR(100) UNIQUE NOT NULL,
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
role VARCHAR(20) DEFAULT 'viewer',
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
last_login TIMESTAMP,
settings JSONB DEFAULT '{}'::jsonb
);
-- Documents table
CREATE TABLE documents (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID REFERENCES users(id) ON DELETE CASCADE,
title VARCHAR(500),
filename VARCHAR(255) NOT NULL,
file_path VARCHAR(500) NOT NULL,
file_hash VARCHAR(64),
file_size BIGINT,
mime_type VARCHAR(100),
content TEXT,
content_vector tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
ocr_status VARCHAR(20) DEFAULT 'pending',
ocr_confidence FLOAT,
metadata JSONB DEFAULT '{}'::jsonb,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
processed_at TIMESTAMP,
source_id UUID REFERENCES sources(id),
CONSTRAINT unique_file_hash UNIQUE(file_hash, user_id)
);
-- Create indexes for performance
CREATE INDEX idx_documents_content_vector ON documents USING gin(content_vector);
CREATE INDEX idx_documents_user_created ON documents(user_id, created_at DESC);
CREATE INDEX idx_documents_metadata ON documents USING gin(metadata jsonb_path_ops);
CREATE INDEX idx_documents_file_hash ON documents(file_hash) WHERE file_hash IS NOT NULL;
```
### Search Optimization
```sql
-- Full-text search function
CREATE OR REPLACE FUNCTION search_documents(
query_text TEXT,
user_id_param UUID,
limit_param INT DEFAULT 20,
offset_param INT DEFAULT 0
) RETURNS TABLE (
id UUID,
title TEXT,
content TEXT,
rank REAL,
snippet TEXT
) AS $$
BEGIN
RETURN QUERY
WITH search_query AS (
SELECT plainto_tsquery('english', query_text) AS q
),
ranked_results AS (
SELECT
d.id,
d.title,
d.content,
ts_rank_cd(d.content_vector, sq.q) AS rank,
ts_headline(
'english',
d.content,
sq.q,
'MaxWords=30, MinWords=15, StartSel=<mark>, StopSel=</mark>'
) AS snippet
FROM documents d, search_query sq
WHERE
d.user_id = user_id_param
AND d.content_vector @@ sq.q
)
SELECT * FROM ranked_results
ORDER BY rank DESC
LIMIT limit_param
OFFSET offset_param;
END;
$$ LANGUAGE plpgsql;
```
## Synchronization Architecture
### WebDAV Sync
```rust
pub struct WebDavSync {
client: WebDavClient,
db: Arc<DbConnection>,
progress: Arc<Mutex<SyncProgress>>,
}
impl WebDavSync {
pub async fn smart_sync(&self) -> Result<SyncResult> {
// 1. Fetch remote file list with ETags
let remote_files = self.client.list_files().await?;
// 2. Compare with local database
let local_files = self.db.get_source_files().await?;
// 3. Determine changes
let changes = self.calculate_changes(&remote_files, &local_files);
// 4. Process changes in batches
for batch in changes.chunks(100) {
self.process_batch(batch).await?;
self.update_progress().await?;
}
// 5. Clean up deleted files
self.process_deletions(&remote_files, &local_files).await?;
Ok(SyncResult {
added: changes.added.len(),
updated: changes.updated.len(),
deleted: changes.deleted.len()
})
}
}
```
### Source Scheduler
```rust
pub struct SourceScheduler {
sources: Arc<RwLock<Vec<Source>>>,
executor: Arc<ThreadPool>,
}
impl SourceScheduler {
pub async fn run(&self) {
loop {
let now = Utc::now();
let sources = self.sources.read().await;
for source in sources.iter() {
if source.should_sync(now) {
let source_clone = source.clone();
self.executor.spawn(async move {
match source_clone.sync().await {
Ok(result) => log::info!("Sync completed: {:?}", result),
Err(e) => log::error!("Sync failed: {}", e),
}
});
}
}
tokio::time::sleep(Duration::from_secs(60)).await;
}
}
}
```
## Performance Optimization
### Connection Pooling
```rust
// Database connection pool configuration
let pool = PgPoolOptions::new()
.max_connections(32)
.min_connections(5)
.connect_timeout(Duration::from_secs(5))
.acquire_timeout(Duration::from_secs(10))
.idle_timeout(Duration::from_secs(600))
.max_lifetime(Duration::from_secs(1800))
.connect(&database_url)
.await?;
```
### Caching Strategy
```rust
// Multi-level caching
pub struct CacheManager {
l1_cache: Arc<DashMap<String, CachedItem>>, // In-memory
l2_cache: Option<RedisClient>, // Redis (optional)
}
impl CacheManager {
pub async fn get<T: DeserializeOwned>(&self, key: &str) -> Option<T> {
// Check L1 cache
if let Some(item) = self.l1_cache.get(key) {
if !item.is_expired() {
return Some(item.value.clone());
}
}
// Check L2 cache
if let Some(redis) = &self.l2_cache {
if let Ok(value) = redis.get(key).await {
self.l1_cache.insert(key.to_string(), value.clone());
return Some(value);
}
}
None
}
}
```
### Batch Processing
```rust
// Batch document processing
pub async fn batch_process_documents(
documents: Vec<Document>,
batch_size: usize,
) -> Result<Vec<ProcessResult>> {
let semaphore = Arc::new(Semaphore::new(batch_size));
let mut tasks = Vec::new();
for doc in documents {
let permit = semaphore.clone().acquire_owned().await?;
let task = tokio::spawn(async move {
let result = process_document(doc).await;
drop(permit);
result
});
tasks.push(task);
}
let results = futures::future::join_all(tasks).await;
Ok(results.into_iter().filter_map(Result::ok).collect())
}
```
## Security Architecture
### Authentication Flow
```
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Client │─────►│ API │─────►│ Auth │
└─────────┘ └─────────┘ └─────────┘
│ │ │
│ POST /login │ Validate │
│ {user,pass} │ Credentials │
│ │ │
│◄───────────────┼─────────────────┤
│ JWT Token │ Generate │
│ │ Token │
│ │ │
│ GET /api/* │ Verify │
│ Auth: Bearer │ JWT │
│ │ │
│◄───────────────┼─────────────────┤
│ API Response │ Authorized │
```
### Permission Model
```rust
// Role-based access control
#[derive(Debug, Clone, PartialEq)]
pub enum Role {
Admin, // Full system access
Editor, // Create, read, update, delete own documents
Viewer, // Read-only access to own documents
}
impl Role {
pub fn can_upload(&self) -> bool {
matches!(self, Role::Admin | Role::Editor)
}
pub fn can_delete(&self) -> bool {
matches!(self, Role::Admin | Role::Editor)
}
pub fn can_manage_users(&self) -> bool {
matches!(self, Role::Admin)
}
pub fn can_configure_system(&self) -> bool {
matches!(self, Role::Admin)
}
}
```
## Scalability Considerations
### Single-Instance Optimization
Since Readur is a single-instance application, scaling is achieved through:
1. **Vertical Scaling**: Increase CPU, RAM, and storage on the single server
2. **Storage Offloading**: Use S3 or compatible object storage
3. **Database Optimization**: Tune PostgreSQL for better performance
4. **Queue Management**: Optimize OCR queue processing
```yaml
# Docker Compose single-instance configuration
version: '3.8'
services:
readur:
image: readur:latest
# Single instance only - do NOT use replicas
deploy:
replicas: 1 # MUST be 1
resources:
limits:
cpus: '4' # Increase for better performance
memory: 4G # Increase for larger workloads
environment:
- DATABASE_URL=postgresql://db:5432/readur
- CONCURRENT_OCR_JOBS=3 # Fixed thread pool
depends_on:
- db
```
### Database Sharding Strategy
```sql
-- Partition documents by user_id for horizontal scaling
CREATE TABLE documents_partition_template (
LIKE documents INCLUDING ALL
) PARTITION BY HASH (user_id);
-- Create partitions
CREATE TABLE documents_part_0 PARTITION OF documents_partition_template
FOR VALUES WITH (modulus 4, remainder 0);
CREATE TABLE documents_part_1 PARTITION OF documents_partition_template
FOR VALUES WITH (modulus 4, remainder 1);
CREATE TABLE documents_part_2 PARTITION OF documents_partition_template
FOR VALUES WITH (modulus 4, remainder 2);
CREATE TABLE documents_part_3 PARTITION OF documents_partition_template
FOR VALUES WITH (modulus 4, remainder 3);
```
## Monitoring and Observability
### Metrics Collection
```rust
// Prometheus metrics
lazy_static! {
static ref HTTP_REQUESTS: IntCounterVec = register_int_counter_vec!(
"http_requests_total",
"Total HTTP requests",
&["method", "endpoint", "status"]
).unwrap();
static ref OCR_PROCESSING_TIME: HistogramVec = register_histogram_vec!(
"ocr_processing_duration_seconds",
"OCR processing time",
&["language", "status"]
).unwrap();
static ref ACTIVE_USERS: IntGauge = register_int_gauge!(
"active_users_total",
"Number of active users"
).unwrap();
}
```
### Distributed Tracing
```rust
// OpenTelemetry integration
use opentelemetry::trace::Tracer;
pub async fn process_document_traced(doc: Document) -> Result<()> {
let tracer = opentelemetry::global::tracer("readur");
let span = tracer.start("process_document");
let cx = Context::current_with_span(span);
// Trace document loading
let _load_span = tracer.start_with_context("load_document", &cx);
let file_data = load_file(&doc.file_path).await?;
// Trace OCR processing
let _ocr_span = tracer.start_with_context("ocr_processing", &cx);
let text = extract_text(&file_data).await?;
// Trace database update
let _db_span = tracer.start_with_context("update_database", &cx);
update_document_content(&doc.id, &text).await?;
Ok(())
}
```
## Deployment Architecture
### Kubernetes Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: readur
spec:
replicas: 3
selector:
matchLabels:
app: readur
template:
metadata:
labels:
app: readur
spec:
containers:
- name: readur
image: readur:latest
ports:
- containerPort: 8080
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: readur-secrets
key: database-url
```
## Development Workflow
### Local Development Setup
```bash
# Development environment
docker-compose -f docker-compose.dev.yml up -d
# Database migrations
cargo run --bin migrate
# Run with hot reload
cargo watch -x run
# Frontend development
cd frontend && npm run dev
```
### Testing Strategy
```rust
// Unit test example
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_document_processing() {
let doc = create_test_document();
let result = process_document(doc).await;
assert!(result.is_ok());
assert_eq!(result.unwrap().status, "completed");
}
#[tokio::test]
async fn test_search_functionality() {
let pool = create_test_pool().await;
seed_test_data(&pool).await;
let results = search_documents("test query", &pool).await;
assert!(!results.is_empty());
}
}
```
## Future Architecture Considerations
### Planned Enhancements
1. **Elasticsearch Integration**: For advanced search capabilities
2. **Machine Learning Pipeline**: For document classification and smart tagging
3. **Microservices Migration**: Separate OCR, search, and storage services
4. **GraphQL API**: Alternative to REST for flexible querying
5. **Event Sourcing**: For audit trail and time-travel debugging
6. **Multi-tenancy**: Support for multiple organizations
### Technology Roadmap
- **Q1 2025**: Redis caching layer
- **Q2 2025**: Elasticsearch integration
- **Q3 2025**: ML-based document classification
- **Q4 2025**: Microservices architecture
## Architecture Decision Records (ADRs)
### ADR-001: Use Rust for Backend
**Status**: Accepted
**Context**: Need high performance and memory safety
**Decision**: Use Rust with Axum framework
**Consequences**: Steep learning curve but excellent performance
### ADR-002: PostgreSQL for Primary Database
**Status**: Accepted
**Context**: Need reliable ACID compliance and full-text search
**Decision**: Use PostgreSQL with built-in FTS
**Consequences**: Single point of failure without replication
### ADR-003: Monolithic Single-Instance Architecture
**Status**: Accepted
**Context**: Simpler architecture, easier deployment and maintenance
**Decision**: Single-instance monolithic application without clustering support
**Consequences**:
- Pros: Simple deployment, no distributed system complexity, easier debugging
- Cons: No high availability, scaling limited to vertical scaling
- Note: This is a deliberate design choice for simplicity and reliability
Reference in New Issue View Git Blame Copy Permalink