d4b57d2ae0
feat(server/client): implement retry functionality for both successful and failed documents
2025-07-02 00:06:47 +00:00
a381cdd12c
feat(webdav): also fix the parser to include directories, and add tests
2025-07-01 22:03:06 +00:00
c1dbd06df2
feat(tests): add unit tests for new webdav functionality
2025-07-01 21:39:31 +00:00
92b21350db
feat(webdav): track directory etags
...
✅ Core Optimizations Implemented
1. 📊 New Database Schema: Added webdav_directories table to track
directory ETags, file counts, and metadata
2. 🔍 Smart Directory Checking: Before deep scans, check directory
ETags with lightweight Depth: 0 PROPFIND requests
3. ΓÜí Skip Unchanged Directories: If directory ETag matches, skip the
entire deep scan
4. 🗂️ N-Depth Subdirectory Tracking: Recursively track all
subdirectories found during scans
5. 🎯 Individual Subdirectory Checks: When parent unchanged, check
each known subdirectory individually
🚀 Performance Benefits
Before: Every sync = Full Depth: infinity scan of entire directory
treeAfter:
- First sync: Full scan + directory tracking setup
- Subsequent syncs: Quick ETag checks → skip unchanged directories
entirely
- Changed directories: Only scan the specific changed subdirectories
📁 How It Works
1. Initial Request: PROPFIND Depth: 0 on /Documents → get directory
ETag
2. Database Check: Compare with stored ETag for /Documents
3. If Unchanged: Check each known subdirectory (/Documents/2024,
/Documents/Archive) individually
4. If Changed: Full recursive scan + update all directory tracking
data
2025-07-01 21:22:16 +00:00
6a23a407bf
feat(client): update swagger ui endpoints
2025-07-01 20:54:45 +00:00
df281f3b26
feat(pdf): implement ocrmypdf to extract text from PDFs
2025-07-01 00:56:48 +00:00
706e20f35c
Merge branch 'main' into feat/debug-page
2025-06-30 17:19:31 -07:00
231f88f038
feat(debug): debug page actually works and does something
2025-07-01 00:15:48 +00:00
0052032772
fix(pdf): resolve PDF wordcount error
2025-07-01 00:10:49 +00:00
830f9d0b38
feat(server): mark documents with 0 words as failed, and fix webdav unit tests
2025-06-30 22:43:25 +00:00
69279344cb
fix(tests): fix documents tests
2025-06-30 21:56:21 +00:00
b38c1fca07
feat(server): fix serialization issues
2025-06-30 19:40:05 +00:00
9e43df2fbe
feat(server/client): add metadata to file view
2025-06-30 19:13:16 +00:00
fef28a33c6
feat(server): continue to try to wrangle the failed and ignored documents
2025-06-29 23:27:51 +00:00
fbf89c213d
fix(tests): resolve a whole lot of test issues
2025-06-28 22:50:40 +00:00
edd0c7514f
fix(server): resolve compilation errors in constraint_validation.rs
2025-06-28 22:04:01 +00:00
97fa50c1b5
feat(server/client): resolve failing tests
2025-06-28 21:21:05 +00:00
84577806ef
feat(server/client): add failed_documents table to handle failures, and move logic of failures
2025-06-28 20:52:58 +00:00
fce56b660b
Merge pull request #72 from readur/feat/better-db-tests
...
feat(tests): add regression tests and better sql type safety tests
2025-06-28 12:43:52 -07:00
f4adafe2bd
feat(tests): add regression tests and better sql type safety tests
2025-06-28 19:25:15 +00:00
9be70dc245
feat(swagger): add missing oidc endpoints into swagger ui
2025-06-28 19:19:48 +00:00
099f4853a7
fix(server): resolve incorrect db type
2025-06-28 18:41:48 +00:00
fe1deb1e9d
fix(server): resolve compilation issues from queue.rs
2025-06-28 18:15:55 +00:00
2d04f0094a
fix(ocr_status): populate the ocr queue with pending jobs and add easy 'retry' button
2025-06-28 18:08:00 +00:00
5aae560d7e
Merge pull request #69 from readur/fix/ocr-confidence-1
...
fix(server/client): fix incorrect OCR measurements
2025-06-28 09:53:56 -07:00
9079529eb5
feat(tests): create generic migration tests
2025-06-28 16:38:12 +00:00
582617ab88
fix(server/client): fix incorrect OCR measurements
2025-06-27 20:23:59 -07:00
cc0d647590
fix(server): resolve compilation issue in IgnoredFilesQuery
2025-06-28 01:01:51 +00:00
0b8dbfb8d9
feat(server/client): easily undelete ignored files, if the user wishes to do so
2025-06-28 00:37:49 +00:00
2c6bd92bf4
fix(server): fix unclosed delimiter
2025-06-27 22:51:02 +00:00
929f27eaa9
Merge branch 'main' into feat/delete-low-confidence-documents
2025-06-27 15:17:50 -07:00
aacfc96825
feat(server/client): implement button deleting low confidence documents (e.g. documents that have no text)
2025-06-27 22:16:38 +00:00
a75fca0c28
feat(client/server): add a new badge for each source that shows the number of documents stored from each source
2025-06-27 21:32:50 +00:00
341a91e1a7
fix(tests): move oidc tests to correct folder
2025-06-27 19:33:58 +00:00
57bb0ccd2c
fix(server): resolve broken imports on tests and test helpers
2025-06-27 18:46:41 +00:00
9a8bf72ff7
feat(server): reorganize components into their own modules and fix imports
2025-06-27 18:27:42 +00:00
b095cb951f
Merge pull request #55 from readur/feat/oidc-setup
...
feat(server): set up oidc system and migrations
2025-06-27 10:48:28 -07:00
0b6d96df03
fix(tests): resolve last OIDC test issues
2025-06-27 17:32:33 +00:00
12cdd0ffd6
fix(tests): resolve some difficult race conditions in test
2025-06-27 05:08:12 +00:00
3c5b7c7dfb
feat(oidc): fix oidc, tests, and everything in between
2025-06-27 05:03:27 +00:00
51907f81f2
fix(metrics): fix broken prometheus metrics
2025-06-26 22:14:42 +00:00
269ba4d46a
Merge pull request #56 from readur/fix/pdf-thumbnail-generation
...
feat(server): actually render PDF thumbnails
2025-06-26 14:14:25 -07:00
e626f3a131
feat(metrics): add more prometheus metrics, and create grafana dashboard
2025-06-26 21:14:00 +00:00
075657899f
feat(server): use poppler for pdf image generation
2025-06-26 20:39:42 +00:00
a94acd7ffe
feat(server): actually render PDF thumbnails?
2025-06-26 20:25:52 +00:00
e9496b921e
feat(server): set up oidc system and migrations
2025-06-26 18:52:57 +00:00
70451d728f
Merge pull request #46 from readur/fix/catch-pdf-extract-errors
...
fix(server): catch pdf-extract spammy logs
2025-06-25 21:35:09 -07:00
715b94ec66
feat(swagger): add a ton of docstrings to functions
2025-06-25 23:58:37 +00:00
20b90e92d3
feat(swagger): add missing endpoints to swagger-ui
2025-06-25 23:47:27 +00:00
40afb5ade5
fix(server): catch pdf-extract spammy logs
2025-06-25 23:26:11 +00:00
perf3ct