Merged memdb with choredb #543
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit deletes the outdated `knowledge.rs` and `mod.rs` files under `src/agent_db`, along with the removal of `agent_db` submodule references. Refactored functionalities are now managed within the newly added `memdb` module. The `memdb` module streamlines database interactions and functions previously managed under `agent_db`. Functions related to pub/sub notifications, vectorization tasks, and memory management are consolidated into `memdb` and its submodules for improved modularity and maintenance efficiency. This includes redefining database schemas, triggers, and event handling to support ongoing system operations and future development.
- Deleted `db_schema_20241102.rs` as it is no longer needed. - Simplified `db_pubsub.rs` by removing unused imports and functions. - Updated pubsub channel names in multiple files to match table names. - Added error handling for chore and chore event settings in `db_chore.rs`. - Refactored trigger setup in `db_schema.rs` to include new chore event triggers.
- Remove `vdb_thread.rs` and migrate its code to a dedicated `vectorizer_service.rs`. - Update module imports to reflect the new organization, removing `vdb_thread` and adding `vectorizer_service`. - Simplify vectorization initialization by directly using `FileVectorizerService` with centralized functions for enqueuing files and memories. - Improve error handling during vectorization processes by avoiding panics and using appropriate logging. - Streamline `VecDb` initialization, background tasks, and file enqueuing logic by better organizing responsibilities between `VecDb` and `FileVectorizerService`. - Ensure `memdb` is initialized within contexts where it is needed, avoiding potential runtime errors. - Retain vectorization functionalities while ensuring modular code structure for easier maintenance and clearer API usage. - Adjust various sections across files in the repo to reflect changes in the vectorization strategy.
…horedb # Conflicts: # refact-agent/engine/src/knowledge.rs # refact-agent/engine/src/memdb/db_init.rs # refact-agent/engine/src/vecdb/mod.rs # refact-agent/engine/src/vecdb/vdb_highlev.rs
- Replace `vec_db_module` with `vectorizer_service` across files for clarity. - Adjust the logic to handle vectorizer service initialization and usage. - Ensure background tasks for vectorizer service are started correctly. - Update import and usage of `memories_add` function from `memdb` module. - Simplify configuration parameters in vector database initialization functions.
# Conflicts: # refact-agent/engine/src/http/routers/v1.rs
- Consolidate vectorization logic by eliminating feature flag checks for vecdb and refactoring related functions. - Use consistent naming for VectorizerService and VecDb across the codebase. - Enhance concurrency safety by using appropriate Arc and Mutex types consistently across modules. - Simplify control flow for various components to manage memory and vectorization tasks more efficiently. - Optimize initialization and background task startup for memory database (memdb) and vector database (vecdb), ensuring tasks are correctly enqueued and processed. - Remove unused index creation code from memdb schema. - Ensure consistent use of async/await for asynchronous operations. - Refactor duplicated code for initializing and checking services into shared utility functions where possible.
- Remove unnecessary memdb initialization checks and related error handling code to streamline the codebase. - Refactor `vecdb` initialization logic, renaming `vecdb_background_reload` to `vecdb_init` to align with its functionality. - Eliminate redundancy by removing `vecdb_constants` from memdb structures, consolidating constant access. - Clean up background task management by removing obsolete components related to `vecdb`. - Update relevant sections to directly access cloned versions of `memdb` without additional option handling. - Optimize the shutdown process by making `memdb_sleeping_point` a direct part of the graceful shutdown procedure.
- Remove conditional VecDb initialization from 'global_context.rs'. - Modify 'vecdb_init.rs' to return a BackgroundTasksHolder. - Add vectorization tasks handling in VecDb initialization. - Update 'background_tasks.rs' to include vecdb background reload task. - Change visibility of 'MessageToVecdbThread' in 'vectorizer_service.rs'. - Adjust 'vectorizer_service_init_and_start' to return task handles. - Refactor error handling in 'vecdb.rs' for clarity. - Simplify vector database reloading logic in 'vdb_highlev.rs' and manage background tasks.
- Rename and encapsulate low-level SQL update and insert operations within their respective set functions. - Remove redundant TODO comments and improve code organization. - Clean up and standardize redundant clone operations. - Ensure consistent formatting and indentation in SQL string literals.
…module - Extract `_cthread_set_lowlevel` logic into `cthread_set_lowlevel` function for better clarity. - Move `merge_json` function from `mod.rs` to a new `db_utils.rs` module, improving code organization. - Update imports of `merge_json` across relevant files to use the new location. - Enhance error logging in subscription handling functions for better debugging.
Correct 'subsription' to 'subscription' in function names and calls. Update field names from 'pubevent_json' to 'pubevent_obj_json' and 'pubevent_memid' to 'pubevent_obj_id'. Introduce tracing for error handling during statement preparation.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.