fix(tests): resolve issue with new url management test
This commit is contained in:
parent
a4e85de9e3
commit
1c1e869362
Binary file not shown.
|
|
@ -1,130 +0,0 @@
|
||||||
## Summary
|
|
||||||
|
|
||||||
This PR implements comprehensive smart sync functionality for WebDAV sources and fixes critical path handling issues that were causing 404 errors with Nextcloud servers.
|
|
||||||
|
|
||||||
## Key Features
|
|
||||||
|
|
||||||
### 🧠 Smart Sync Implementation
|
|
||||||
- **Directory ETag Tracking**: Tracks ETags for all directories at infinite depth to detect changes
|
|
||||||
- **Bulk Database Operations**: Fetches all directory ETags in single query to avoid database round trips
|
|
||||||
- **Intelligent Sync Strategies**:
|
|
||||||
- **SkipSync**: No changes detected, sync skipped entirely
|
|
||||||
- **TargetedScan**: Scans only changed directories for efficiency
|
|
||||||
- **FullDeepScan**: Complete rescan when many changes detected or first-time sync
|
|
||||||
- **Smart Decision Making**: Automatically chooses optimal strategy based on change scope
|
|
||||||
|
|
||||||
### 🔧 WebDAV Path Management Overhaul
|
|
||||||
- **Centralized URL Management**: New `url_management.rs` module handles all WebDAV path operations
|
|
||||||
- **Fixed Nextcloud URL Doubling**: Resolved critical issue where "remote.php" was incorrectly appended causing 404s
|
|
||||||
- **Multi-Server Support**: Proper path handling for Nextcloud, ownCloud, and generic WebDAV servers
|
|
||||||
- **Path Field Migration**: Added `relative_path` and `full_path` fields while maintaining backward compatibility
|
|
||||||
|
|
||||||
### 📊 Enhanced Data Model
|
|
||||||
```rust
|
|
||||||
pub struct FileIngestionInfo {
|
|
||||||
/// Clean relative path from WebDAV root (e.g., "/Photos/image.jpg")
|
|
||||||
pub relative_path: String,
|
|
||||||
/// Full WebDAV path as returned by server (e.g., "/remote.php/dav/files/user/Photos/image.jpg")
|
|
||||||
pub full_path: String,
|
|
||||||
/// Legacy field - deprecated, use relative_path instead
|
|
||||||
#[deprecated(note = "Use relative_path instead for new code")]
|
|
||||||
pub path: String,
|
|
||||||
// ... other fields
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Technical Implementation
|
|
||||||
|
|
||||||
### Smart Sync Service Architecture
|
|
||||||
- **SmartSyncService**: Main service for intelligent sync evaluation and execution
|
|
||||||
- **SmartSyncDecision**: Enum determining whether sync is needed
|
|
||||||
- **SmartSyncStrategy**: Enum defining how sync should be performed
|
|
||||||
- **Bulk Directory Fetching**: Single database query for all directory ETags
|
|
||||||
- **Recursive Directory Tracking**: Tracks subdirectories at all depth levels
|
|
||||||
|
|
||||||
### URL Management System
|
|
||||||
- **Server-Specific Logic**: Handles different WebDAV server path formats
|
|
||||||
- **Path Conversion**: Converts between full WebDAV paths and relative paths
|
|
||||||
- **URL Construction**: Builds correct URLs for file operations
|
|
||||||
- **Backward Compatibility**: Maintains existing functionality during migration
|
|
||||||
|
|
||||||
## Problem Solved
|
|
||||||
|
|
||||||
### Original Issue
|
|
||||||
User discovered that only one folder's ETag was being tracked in the database instead of all subfolders in the WebDAV directory hierarchy. This meant:
|
|
||||||
- Only root directory changes were detected
|
|
||||||
- Subdirectory changes were missed
|
|
||||||
- No performance optimization for unchanged directory trees
|
|
||||||
- Unnecessary full scans on every sync
|
|
||||||
|
|
||||||
### WebDAV Path Issue
|
|
||||||
Nextcloud users experienced 404 errors due to URL doubling:
|
|
||||||
- **Before**: `https://server.com/remote.php/dav/files/user/remote.php/dav/files/user/Photos/image.jpg`
|
|
||||||
- **After**: `https://server.com/remote.php/dav/files/user/Photos/image.jpg`
|
|
||||||
|
|
||||||
## Changes Made
|
|
||||||
|
|
||||||
### Core Implementation
|
|
||||||
- ✅ Created `SmartSyncService` with comprehensive directory ETag tracking
|
|
||||||
- ✅ Implemented bulk database operations for performance
|
|
||||||
- ✅ Added smart sync as default behavior (not optional)
|
|
||||||
- ✅ Created centralized `url_management.rs` module
|
|
||||||
- ✅ Updated XML parser to use new path management
|
|
||||||
- ✅ Migrated FileIngestionInfo to new field structure
|
|
||||||
|
|
||||||
### Testing Infrastructure
|
|
||||||
- ✅ **22+ test files updated** with new field requirements
|
|
||||||
- ✅ **Comprehensive test coverage** for all smart sync scenarios
|
|
||||||
- ✅ **Integration tests** for first-time sync, directory changes, deep scans
|
|
||||||
- ✅ **Unit tests** for decision logic, ETag comparison, strategy selection
|
|
||||||
- ✅ **Path handling tests** to prevent regression of URL doubling issue
|
|
||||||
- ✅ **Database connection pool** fixes for test environment
|
|
||||||
|
|
||||||
### Backward Compatibility
|
|
||||||
- ✅ Deprecated `path` field maintained with warnings
|
|
||||||
- ✅ Existing code continues to work during migration
|
|
||||||
- ✅ Clean migration path to new field structure
|
|
||||||
|
|
||||||
## Test Results
|
|
||||||
|
|
||||||
```bash
|
|
||||||
✅ All library tests compile: cargo test --lib --no-run
|
|
||||||
✅ Integration tests compile successfully
|
|
||||||
✅ No compilation errors (only expected deprecation warnings)
|
|
||||||
✅ Comprehensive test coverage for all scenarios
|
|
||||||
```
|
|
||||||
|
|
||||||
## Performance Impact
|
|
||||||
|
|
||||||
### Before
|
|
||||||
- Database query for each directory check
|
|
||||||
- Full scan on every sync regardless of changes
|
|
||||||
- Inefficient for large directory structures
|
|
||||||
- URL construction errors causing failed requests
|
|
||||||
|
|
||||||
### After
|
|
||||||
- Single bulk query for all directory ETags
|
|
||||||
- Smart sync skips unchanged directory trees
|
|
||||||
- Targeted scans for minimal changes
|
|
||||||
- Correct URL construction for all server types
|
|
||||||
- Significant performance improvement for large WebDAV folders
|
|
||||||
|
|
||||||
## Migration Notes
|
|
||||||
|
|
||||||
- **Smart sync is now the default behavior** (not optional)
|
|
||||||
- **Deep scans reset all directory ETags** at all levels for fresh baselines
|
|
||||||
- **Path field deprecation** - new code should use `relative_path` and `full_path`
|
|
||||||
- **URL management centralized** - prevents future path handling issues
|
|
||||||
|
|
||||||
## Future Benefits
|
|
||||||
|
|
||||||
This implementation provides the foundation for:
|
|
||||||
- More efficient WebDAV synchronization
|
|
||||||
- Better support for large directory structures
|
|
||||||
- Reliable path handling across different WebDAV servers
|
|
||||||
- Extensible smart sync strategies
|
|
||||||
- Improved user experience with faster syncs
|
|
||||||
|
|
||||||
🤖 Generated with [Claude Code](https://claude.ai/code)
|
|
||||||
|
|
||||||
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
||||||
Loading…
Reference in New Issue