42bdda9476
feat(client): show new file metadata fields on the client
2025-07-10 21:07:35 +00:00
db41b15609
feat(client): show more fields for Documents
2025-07-10 21:02:15 +00:00
c90c7201e5
feat(server/client): make sure that the documents endpoint isn't broken
2025-07-10 19:57:25 +00:00
c3af038f02
fix(tests): no way, all the integration tests pass now
2025-07-10 01:38:55 +00:00
22e196d554
fix(tests): resolve integration test response format
2025-07-09 20:10:36 +00:00
43248374f1
fix(server/client): rename document_id to id in DocumentUploadResponse, again
2025-07-09 01:40:50 +00:00
ea94dff8ba
fix(stats): create new get_queue_statistics function to avoid conflicts
2025-07-09 00:27:43 +00:00
9846ea67cb
debug(tests): add some debug lines to see why CI is upset
2025-07-08 22:32:32 +00:00
36b5330622
fix(stats): try to fix the stats extraction, again
2025-07-08 21:18:21 +00:00
8b1cf027a3
fix(server): resolve incorrect document failure titles
2025-07-08 20:24:52 +00:00
bb2bf890d3
fix(stats): try to fix stats export, again again again
2025-07-08 20:21:46 +00:00
f17b610501
fix(stats): try to fix stats export, again again
2025-07-08 20:16:33 +00:00
d51d1f1c78
fix(stats): try to fix stats export, again
2025-07-08 20:03:55 +00:00
05a0355796
fix(tests): fix the crazy metrics collection issue
2025-07-08 16:52:23 +00:00
7d48480cd6
fix(tests): and resolve missing endpoint
2025-07-08 04:37:33 +00:00
72ad31453f
debug(tests): add some debug lines to see why CI is upset
2025-07-08 00:03:30 +00:00
bf2162ad89
fix(web_upload): resolve issue that caused files that were uploaded via the web, to not be added to the queue
2025-07-07 19:28:08 +00:00
2a68c0a066
feat(server): implement better error checking for sources
2025-07-07 19:10:45 +00:00
ab610bc33f
feat(webdav): resolve failing etag unit tests
2025-07-05 21:16:15 +00:00
b184f0f68f
feat(webdav): resolve failing etag unit tests
2025-07-05 19:47:21 +00:00
7c9b29676c
feat(tests): move integration and unit tests to correct locations
2025-07-04 19:37:43 +00:00
fd26e1a0ae
fix(tests): binary and tests at least compile now
2025-07-04 19:07:53 +00:00
8c17578c8f
Merge pull request #96 from readur/feat/deduplicate-test-utils-1
...
feat(tests): deduplicate test functionalities
2025-07-04 09:12:40 -07:00
a19b6fb60b
fix(server): fix axum groups
2025-07-04 03:07:28 +00:00
0b0ffd1dbf
fix(server): resolve compilation errors due to splitting up the large files
2025-07-04 03:06:29 +00:00
1b984a12c2
fix(server): resolve type issues and functions for compilation issues
2025-07-04 00:53:32 +00:00
607017e927
fix(tests): resolve broken test utils
2025-07-04 00:31:53 +00:00
6bbf69842c
fix(server): resolve import issues
2025-07-03 23:58:11 +00:00
5daa1aedef
feat(dev): break up the large sources.rs file into smaller ones
2025-07-03 23:44:49 +00:00
220d6612a7
feat(dev): break up the large documents.rs file, again
2025-07-03 23:33:53 +00:00
6547130fb1
feat(dev): also break up the large webdav_service.rs file into smaller ones
2025-07-03 19:57:31 +00:00
2b7d901b9d
feat(dev): break up the large documents.rs file
2025-07-03 19:47:31 +00:00
2a6f20b0d3
feat(dev): split up large models.rs file to smaller ones
2025-07-03 19:35:36 +00:00
bdf4f5f8fe
feat(ocr): add even more about the multiple ocr languages
2025-07-03 19:20:19 +00:00
465d1dba4f
feat(tests): try to deduplicate test code even more
2025-07-03 19:17:33 +00:00
14b29872e8
feat(tests): deduplicate tests too
2025-07-03 17:21:39 +00:00
720351122a
feat(webdav): add validation statuses to sources
2025-07-03 14:03:26 +00:00
c0835f436f
feat(webdav): also add some crazy source automatic validation
2025-07-03 05:26:36 +00:00
99cbb9caee
feat(webdav): make sure to have scanned all subdirectories
2025-07-03 05:02:17 +00:00
42c7edd9df
feat(webdav): gracefully recover webdav from stops/crashes
2025-07-03 04:45:25 +00:00
915fe92993
feat(webdav): also set up deep scanning button and fix unit tests
2025-07-03 04:24:26 +00:00
e23a0e4c63
feat(webdav): directory etag smart checking and all that
2025-07-03 00:26:56 +00:00
0ef233c3cc
fix(tests): resolve compilation error in tests and source scheduler
2025-07-02 23:49:46 +00:00
edcf2be71b
fix(webdav): resolve issue with webdav subdirectories not being discovered
2025-07-02 23:37:39 +00:00
1c131d8fc3
fix(tests): resolve silly new ocr retry tests
2025-07-02 22:51:09 +00:00
f1fc2a0105
fix(server): resolve issues with the retry ocr tests
2025-07-02 22:47:51 +00:00
bfef816848
fix(server): resolve ocr test functionality failing due to db trigger
2025-07-02 22:38:13 +00:00
4f0497ba74
feat(tests): fix ocr_retry issues in tests
2025-07-02 21:48:01 +00:00
dec4551fbd
feat(tests): fix ocr_retry issues in tests
2025-07-02 21:30:36 +00:00
e15aa801c6
feat(tests): fix ocr_retry issues in tests
2025-07-02 18:48:26 +00:00
dd963d0ecc
feat(server): allow also completed documents to be retried
2025-07-02 18:15:41 +00:00
8ed8701d5b
feat(server): implement DEBUG environment variable
2025-07-02 17:57:57 +00:00
0f3cb12c7a
fix(server): resolve NUMERIC db type and f64 rust type
2025-07-02 02:26:11 +00:00
e4faf2cfd2
feat(server/client): implement retry functionality for both successful and failed documents
2025-07-02 00:06:47 +00:00
c242a84326
feat(webdav): also fix the parser to include directories, and add tests
2025-07-01 22:03:06 +00:00
590cad3197
feat(tests): add unit tests for new webdav functionality
2025-07-01 21:39:31 +00:00
fdc240fa5b
feat(webdav): track directory etags
...
✅ Core Optimizations Implemented
1. 📊 New Database Schema: Added webdav_directories table to track
directory ETags, file counts, and metadata
2. 🔍 Smart Directory Checking: Before deep scans, check directory
ETags with lightweight Depth: 0 PROPFIND requests
3. ΓÜí Skip Unchanged Directories: If directory ETag matches, skip the
entire deep scan
4. 🗂️ N-Depth Subdirectory Tracking: Recursively track all
subdirectories found during scans
5. 🎯 Individual Subdirectory Checks: When parent unchanged, check
each known subdirectory individually
🚀 Performance Benefits
Before: Every sync = Full Depth: infinity scan of entire directory
treeAfter:
- First sync: Full scan + directory tracking setup
- Subsequent syncs: Quick ETag checks → skip unchanged directories
entirely
- Changed directories: Only scan the specific changed subdirectories
📁 How It Works
1. Initial Request: PROPFIND Depth: 0 on /Documents → get directory
ETag
2. Database Check: Compare with stored ETag for /Documents
3. If Unchanged: Check each known subdirectory (/Documents/2024,
/Documents/Archive) individually
4. If Changed: Full recursive scan + update all directory tracking
data
2025-07-01 21:22:16 +00:00
a2ea446e16
feat(client): update swagger ui endpoints
2025-07-01 20:54:45 +00:00
a88f387aeb
Merge branch 'main' into feat/multiple-ocr-languages
2025-07-01 11:53:42 -07:00
f7018575d8
feat(pdf): implement ocrmypdf to extract text from PDFs
2025-07-01 00:56:48 +00:00
59e80a1b92
Merge branch 'main' into feat/debug-page
2025-06-30 17:19:31 -07:00
2a1eeeda50
feat(debug): debug page actually works and does something
2025-07-01 00:15:48 +00:00
f26ab1e367
fix(pdf): resolve PDF wordcount error
2025-07-01 00:10:49 +00:00
dd90e48fd2
feat(server): mark documents with 0 words as failed, and fix webdav unit tests
2025-06-30 22:43:25 +00:00
bf073132a1
fix(tests): fix documents tests
2025-06-30 21:56:21 +00:00
b344b69da7
feat(server): fix serialization issues
2025-06-30 19:40:05 +00:00
d9b695f0bd
feat(server/client): add metadata to file view
2025-06-30 19:13:16 +00:00
5f10a8b82c
feat(server): continue to try to wrangle the failed and ignored documents
2025-06-29 23:27:51 +00:00
8d1a886139
fix(tests): resolve compilation error in the multiple OCR functionality
2025-06-29 23:21:42 +00:00
e0b0f49ba2
feat(tests): implement and update tests for multiple OCR languages
2025-06-29 23:03:37 +00:00
b4ddf034b0
feat(server/client): support multiple OCR languages
2025-06-29 22:51:06 +00:00
28a7e4eb45
fix(tests): resolve a whole lot of test issues
2025-06-28 22:50:40 +00:00
2b29032a42
fix(server): resolve compilation errors in constraint_validation.rs
2025-06-28 22:04:01 +00:00
df254d59e3
feat(server/client): resolve failing tests
2025-06-28 21:21:05 +00:00
34bc207e39
feat(server/client): add failed_documents table to handle failures, and move logic of failures
2025-06-28 20:52:58 +00:00
881e4c5a8e
Merge pull request #72 from readur/feat/better-db-tests
...
feat(tests): add regression tests and better sql type safety tests
2025-06-28 12:43:52 -07:00
25d6ecce6c
feat(tests): add regression tests and better sql type safety tests
2025-06-28 19:25:15 +00:00
e2633d7162
feat(swagger): add missing oidc endpoints into swagger ui
2025-06-28 19:19:48 +00:00
a792d2e6fd
fix(server): resolve incorrect db type
2025-06-28 18:41:48 +00:00
d18daa2c17
fix(server): resolve compilation issues from queue.rs
2025-06-28 18:15:55 +00:00
6dd580fa2f
fix(ocr_status): populate the ocr queue with pending jobs and add easy 'retry' button
2025-06-28 18:08:00 +00:00
a314f64ca9
Merge pull request #69 from readur/fix/ocr-confidence-1
...
fix(server/client): fix incorrect OCR measurements
2025-06-28 09:53:56 -07:00
7247f74456
feat(tests): create generic migration tests
2025-06-28 16:38:12 +00:00
7f69cd2e5f
fix(server/client): fix incorrect OCR measurements
2025-06-27 20:23:59 -07:00
09a36bd0fb
fix(server): resolve compilation issue in IgnoredFilesQuery
2025-06-28 01:01:51 +00:00
7a623ca8d6
feat(server/client): easily undelete ignored files, if the user wishes to do so
2025-06-28 00:37:49 +00:00
6983469eff
fix(server): fix unclosed delimiter
2025-06-27 22:51:02 +00:00
a8c6660450
Merge branch 'main' into feat/delete-low-confidence-documents
2025-06-27 15:17:50 -07:00
a642eec3ce
feat(server/client): implement button deleting low confidence documents (e.g. documents that have no text)
2025-06-27 22:16:38 +00:00
fc0324da80
feat(client/server): add a new badge for each source that shows the number of documents stored from each source
2025-06-27 21:32:50 +00:00
9073006004
fix(tests): move oidc tests to correct folder
2025-06-27 19:33:58 +00:00
60dee61f16
fix(server): resolve broken imports on tests and test helpers
2025-06-27 18:46:41 +00:00
cdad6477ed
feat(server): reorganize components into their own modules and fix imports
2025-06-27 18:27:42 +00:00
c8f4886fc6
Merge pull request #55 from readur/feat/oidc-setup
...
feat(server): set up oidc system and migrations
2025-06-27 10:48:28 -07:00
af3aab3cda
fix(tests): resolve last OIDC test issues
2025-06-27 17:32:33 +00:00
98479325e3
fix(tests): resolve some difficult race conditions in test
2025-06-27 05:08:12 +00:00
72708a05f3
feat(oidc): fix oidc, tests, and everything in between
2025-06-27 05:03:27 +00:00
2c39f96dcf
fix(metrics): fix broken prometheus metrics
2025-06-26 22:14:42 +00:00
1c02dd480e
Merge pull request #56 from readur/fix/pdf-thumbnail-generation
...
feat(server): actually render PDF thumbnails
2025-06-26 14:14:25 -07:00
daac9599b8
feat(metrics): add more prometheus metrics, and create grafana dashboard
2025-06-26 21:14:00 +00:00
perf3ct