From f76f487211d3859edf2d14f0b11812811df3a758 Mon Sep 17 00:00:00 2001 From: Chirag Chandrashekhar Date: Wed, 25 Feb 2026 19:15:48 +0530 Subject: [PATCH] Performance/file parse and mount (#6975) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Refactor: optimize collection updates with batch processing - Introduced BatchAggregator to handle IPC events in batches, reducing Redux dispatch overhead during collection mounting. - Updated collection watcher to utilize batch processing for adding files and directories, improving UI performance. - Implemented ParsedFileCacheStore using LMDB for efficient caching of parsed file content, enhancing loading speed and reducing redundant parsing. - Adjusted collection slice to support batch addition of items, minimizing re-renders and improving state management. - Updated relevant components to reflect changes in loading states and collection data handling. * feat: add cache management to preferences - Introduced a new Cache component in the Preferences section to display cache statistics and allow users to purge the cache. - Implemented IPC handlers for fetching cache stats and purging the cache in the Electron main process. - Added styled components for better UI presentation of cache information. - Updated Preferences component to include a new tab for cache management. * fix: update package-lock.json to change 'devOptional' to 'dev' for several Babel dependencies * refactor: update batch aggregation parameters for improved performance - Increased DISPATCH_INTERVAL_MS from 150ms to 200ms for better timing control. - Adjusted MAX_BATCH_SIZE from 200 to 300 items to enhance batch processing efficiency. * feat: enhance collection loading state and improve batch aggregator functionality - Added isLoading property to collections slice to manage loading state during collection operations. - Updated getAggregator function calls in collection-watcher to include collectionUid for better context in batch processing. - Normalized directory path handling in parsed-file-cache to ensure consistent prefix creation for cache keys. * fix: update loading state and transient file handling in collections slice - Changed isLoading property to false during collection initialization for accurate loading state representation. - Introduced isTransient flag for directories and files to differentiate between transient and non-transient items. - Enhanced logic for handling transient directories and files during collection processing to improve state management. * feat: add batch processing support for file additions in task middleware - Implemented a new listener for collectionBatchAddItems to handle batch file additions. - Enhanced task management by checking for pending OPEN_REQUEST tasks that match added files. - Improved tab management by dispatching addTab actions for matching files and removing corresponding tasks from the queue. * feat: enable ASAR packaging and unpacking for LMDB binaries in Electron build configuration - Added ASAR support to the Electron build configuration for the Bruno application. - Specified unpacking rules for LMDB native binaries to ensure proper loading during runtime. * feat: implement parsed file cache using IndexedDB for improved performance - Introduced a new `parsedFileCacheStore` utilizing IndexedDB for caching parsed file data. - Replaced the previous LMDB-based cache implementation to enhance performance and reliability. - Updated IPC handlers to manage cache operations such as get, set, invalidate, and clear. - Integrated the new cache store into various components, ensuring efficient data retrieval and storage. - Added pruning functionality to remove outdated cache entries on startup. * refactor: update collection root and item handling to preserve UIDs - Modified the way collection roots and folder items are assigned by using `mergeRootWithPreservedUids` and `mergeRequestWithPreservedUids` to ensure UIDs are maintained during updates. - This change enhances data integrity when managing collections and their associated files. * refactor: pass mainWindow reference to parsedFileCacheStore initialization - Updated the `initialize` method in `ParsedFileCacheStore` to accept a `mainWindow` parameter, allowing for direct access to the main window instance in IPC handlers. - This change improves the handling of IPC requests by ensuring the correct window context is used for sending messages. * refactor: optimize getStats method in parsedFileCacheStore for performance - Replaced manual counting of total files with a direct count() call for O(1) performance. - Updated the collection counting logic to utilize openKeyCursor with 'nextunique' for improved efficiency in counting unique collection paths. - These changes enhance the performance of the getStats method by reducing the complexity of file and collection counting. * fix: update key generation in parsedFileCache to use newline separator - Changed the key generation logic in `generateKey` from a null character to a newline character for improved readability and consistency in cache keys. * refactor: rename batch-aggregator to collection-tree-batcher and add tests - Rename BatchAggregator class to CollectionTreeBatcher - Rename getAggregator/removeAggregator to getBatcher/removeBatcher - Update imports and variable names in collection-watcher.js - Add backward-compatible aliases for old names - Add 22 unit tests covering all functionality * refactor: update key generation in parsedFileCache to use custom separator - Changed the key generation logic in `generateKey` to use a custom separator (↝) instead of a newline character for improved readability and consistency in cache keys. * fix: add missing reject handler and fix directory prefix collision - Add reject to Promise and pendingRequests in parsed-file-cache-idb.js - Normalize dirPath with trailing separator in invalidateDirectory to prevent false matches (e.g., /foo/bar matching /foo/barley) - Use platform-specific path.sep for cross-platform compatibility * fix: add error handling in parsedFileCache and update window close event - Added a catch block to handle errors in the database promise in parsedFileCache. - Updated the window close event listener in collection-tree-batcher to use `once` for better resource management. * fix: add LRU eviction when IndexedDB quota is exceeded Handle QuotaExceededError in setEntry by automatically evicting the oldest 20% of cache entries and retrying the write operation. * fix: use once instead of on in mock window for batcher tests --------- Co-authored-by: Chirag Chandrashekhar --- packages/bruno-app/jsconfig.json | 3 +- .../CollectionSettings/Overview/Info/index.js | 16 +- .../Preferences/Cache/StyledWrapper.js | 67 ++++ .../src/components/Preferences/Cache/index.js | 89 +++++ .../src/components/Preferences/index.js | 12 +- .../Sidebar/Collections/Collection/index.js | 3 +- packages/bruno-app/src/providers/App/index.js | 2 + .../src/providers/App/useIpcEvents.js | 48 +++ .../providers/App/useParsedFileCacheIpc.js | 60 ++++ .../middlewares/tasks/middleware.js | 53 ++- .../ReduxStore/slices/collections/index.js | 155 +++++++++ .../bruno-app/src/store/parsedFileCache.js | 249 ++++++++++++++ .../src/app/collection-tree-batcher.js | 196 +++++++++++ .../src/app/collection-watcher.js | 131 +++++--- packages/bruno-electron/src/index.js | 4 + .../bruno-electron/src/ipc/preferences.js | 20 ++ .../src/store/parsed-file-cache-idb.js | 157 +++++++++ .../tests/app/collection-tree-batcher.spec.js | 312 ++++++++++++++++++ 18 files changed, 1516 insertions(+), 61 deletions(-) create mode 100644 packages/bruno-app/src/components/Preferences/Cache/StyledWrapper.js create mode 100644 packages/bruno-app/src/components/Preferences/Cache/index.js create mode 100644 packages/bruno-app/src/providers/App/useParsedFileCacheIpc.js create mode 100644 packages/bruno-app/src/store/parsedFileCache.js create mode 100644 packages/bruno-electron/src/app/collection-tree-batcher.js create mode 100644 packages/bruno-electron/src/store/parsed-file-cache-idb.js create mode 100644 packages/bruno-electron/tests/app/collection-tree-batcher.spec.js diff --git a/packages/bruno-app/jsconfig.json b/packages/bruno-app/jsconfig.json index a71bc3138..257b99e10 100644 --- a/packages/bruno-app/jsconfig.json +++ b/packages/bruno-app/jsconfig.json @@ -13,7 +13,8 @@ "api/*": ["src/api/*"], "pageComponents/*": ["src/pageComponents/*"], "providers/*": ["src/providers/*"], - "utils/*": ["src/utils/*"] + "utils/*": ["src/utils/*"], + "store/*": ["src/store/*"] } }, "exclude": ["node_modules", "dist"] diff --git a/packages/bruno-app/src/components/CollectionSettings/Overview/Info/index.js b/packages/bruno-app/src/components/CollectionSettings/Overview/Info/index.js index 888149f78..99051b5f4 100644 --- a/packages/bruno-app/src/components/CollectionSettings/Overview/Info/index.js +++ b/packages/bruno-app/src/components/CollectionSettings/Overview/Info/index.js @@ -1,7 +1,6 @@ -import React from 'react'; +import React, { useMemo } from 'react'; import { getTotalRequestCountInCollection } from 'utils/collections/'; import { IconFolder, IconWorld, IconApi, IconShare, IconBook } from '@tabler/icons'; -import { areItemsLoading, getItemsLoadStats } from 'utils/collections/index'; import { useState } from 'react'; import { useSelector, useDispatch } from 'react-redux'; import ShareCollection from 'components/ShareCollection/index'; @@ -11,10 +10,13 @@ import StyledWrapper from './StyledWrapper'; const Info = ({ collection }) => { const dispatch = useDispatch(); - const totalRequestsInCollection = getTotalRequestCountInCollection(collection); - const isCollectionLoading = areItemsLoading(collection); - const { loading: itemsLoadingCount, total: totalItems } = getItemsLoadStats(collection); + const isCollectionLoading = collection.isLoading; + + const totalRequestsInCollection = useMemo( + () => getTotalRequestCountInCollection(collection), + [collection.items] + ); const [showShareCollectionModal, toggleShowShareCollectionModal] = useState(false); const [showGenerateDocumentationModal, setShowGenerateDocumentationModal] = useState(false); @@ -95,7 +97,9 @@ const Info = ({ collection }) => {
Requests
{ - isCollectionLoading ? `${totalItems - itemsLoadingCount} out of ${totalItems} requests in the collection loaded` : `${totalRequestsInCollection} request${totalRequestsInCollection !== 1 ? 's' : ''} in collection` + isCollectionLoading + ? 'Loading requests...' + : `${totalRequestsInCollection} request${totalRequestsInCollection !== 1 ? 's' : ''} in collection` }
diff --git a/packages/bruno-app/src/components/Preferences/Cache/StyledWrapper.js b/packages/bruno-app/src/components/Preferences/Cache/StyledWrapper.js new file mode 100644 index 000000000..81dd38e70 --- /dev/null +++ b/packages/bruno-app/src/components/Preferences/Cache/StyledWrapper.js @@ -0,0 +1,67 @@ +import styled from 'styled-components'; + +const StyledWrapper = styled.div` + color: ${(props) => props.theme.text}; + + .cache-stats { + padding: 1rem; + border-radius: ${(props) => props.theme.border.radius.md}; + background-color: ${(props) => props.theme.input.bg}; + border: 1px solid ${(props) => props.theme.input.border}; + margin-bottom: 1rem; + } + + .stat-item { + display: flex; + justify-content: space-between; + padding: 0.5rem 0; + border-bottom: 1px solid ${(props) => props.theme.input.border}; + + &:last-child { + border-bottom: none; + } + } + + .stat-label { + font-size: ${(props) => props.theme.font.size.sm}; + color: ${(props) => props.theme.colors.text.muted}; + } + + .stat-value { + font-size: ${(props) => props.theme.font.size.sm}; + font-weight: 500; + } + + .purge-button { + padding: 0.5rem 1rem; + border-radius: ${(props) => props.theme.border.radius.sm}; + font-size: ${(props) => props.theme.font.size.sm}; + cursor: pointer; + background-color: ${(props) => props.theme.input.bg}; + border: 1px solid ${(props) => props.theme.input.border}; + color: ${(props) => props.theme.text}; + + &:hover:not(:disabled) { + border-color: ${(props) => props.theme.input.focusBorder}; + } + + &:disabled { + opacity: 0.5; + cursor: not-allowed; + } + } + + .description { + font-size: ${(props) => props.theme.font.size.sm}; + color: ${(props) => props.theme.colors.text.muted}; + margin-top: 0.5rem; + } + + .section-title { + font-weight: 600; + font-size: 0.875rem; + margin-bottom: 0.75rem; + } +`; + +export default StyledWrapper; diff --git a/packages/bruno-app/src/components/Preferences/Cache/index.js b/packages/bruno-app/src/components/Preferences/Cache/index.js new file mode 100644 index 000000000..a810a2a45 --- /dev/null +++ b/packages/bruno-app/src/components/Preferences/Cache/index.js @@ -0,0 +1,89 @@ +import React, { useState, useEffect, useCallback } from 'react'; +import toast from 'react-hot-toast'; +import StyledWrapper from './StyledWrapper'; + +const Cache = () => { + const [stats, setStats] = useState(null); + const [loading, setLoading] = useState(true); + const [purging, setPurging] = useState(false); + + const fetchStats = useCallback(async () => { + try { + const cacheStats = await window.ipcRenderer.invoke('renderer:get-cache-stats'); + setStats(cacheStats); + } catch (error) { + console.error('Error fetching cache stats:', error); + setStats({ error: error.message }); + } finally { + setLoading(false); + } + }, []); + + useEffect(() => { + fetchStats(); + }, [fetchStats]); + + const handlePurgeCache = async () => { + setPurging(true); + try { + const result = await window.ipcRenderer.invoke('renderer:purge-cache'); + if (result.success) { + toast.success('Cache purged successfully'); + await fetchStats(); + } else { + toast.error(result.error || 'Failed to purge cache'); + } + } catch (error) { + console.error('Error purging cache:', error); + toast.error('Failed to purge cache'); + } finally { + setPurging(false); + } + }; + + return ( + +
Collection Cache
+

+ Bruno caches parsed collection files to improve loading performance. Clearing the cache will cause collections to be fully re-parsed on next load. +

+ +
+ {loading ? ( +
+ Loading... +
+ ) : stats?.error ? ( +
+ Error: {stats.error} +
+ ) : ( + <> +
+ Cached Collections + {stats?.totalCollections ?? 0} +
+
+ Cached Files + {stats?.totalFiles ?? 0} +
+
+ Cache Version + {stats?.version ?? 'N/A'} +
+ + )} +
+ + + + ); +}; + +export default Cache; diff --git a/packages/bruno-app/src/components/Preferences/index.js b/packages/bruno-app/src/components/Preferences/index.js index 547ffd09a..ea7c188bc 100644 --- a/packages/bruno-app/src/components/Preferences/index.js +++ b/packages/bruno-app/src/components/Preferences/index.js @@ -9,7 +9,8 @@ import { IconUserCircle, IconKeyboard, IconZoomQuestion, - IconSquareLetterB + IconSquareLetterB, + IconDatabase } from '@tabler/icons'; import Support from './Support'; @@ -19,6 +20,7 @@ import Proxy from './ProxySettings'; import Display from './Display'; import Keybindings from './Keybindings'; import Beta from './Beta'; +import Cache from './Cache'; import StyledWrapper from './StyledWrapper'; @@ -62,6 +64,10 @@ const Preferences = () => { return ; } + case 'cache': { + return ; + } + case 'support': { return ; } @@ -92,6 +98,10 @@ const Preferences = () => { Keybindings +
setTab('cache')}> + + Cache +
setTab('support')}> Support diff --git a/packages/bruno-app/src/components/Sidebar/Collections/Collection/index.js b/packages/bruno-app/src/components/Sidebar/Collections/Collection/index.js index 90b830137..358d01131 100644 --- a/packages/bruno-app/src/components/Sidebar/Collections/Collection/index.js +++ b/packages/bruno-app/src/components/Sidebar/Collections/Collection/index.js @@ -38,7 +38,6 @@ import { isTabForItemActive } from 'src/selectors/tab'; import RenameCollection from './RenameCollection'; import StyledWrapper from './StyledWrapper'; import CloneCollection from './CloneCollection'; -import { areItemsLoading } from 'utils/collections'; import { scrollToTheActiveTab } from 'utils/tabs'; import ShareCollection from 'components/ShareCollection/index'; import GenerateDocumentation from './GenerateDocumentation'; @@ -62,7 +61,7 @@ const Collection = ({ collection, searchText }) => { const [dropType, setDropType] = useState(null); const [isKeyboardFocused, setIsKeyboardFocused] = useState(false); const dispatch = useDispatch(); - const isLoading = areItemsLoading(collection); + const isLoading = collection.isLoading; const collectionRef = useRef(null); const isCollectionFocused = useSelector(isTabForItemActive({ itemUid: collection.uid })); diff --git a/packages/bruno-app/src/providers/App/index.js b/packages/bruno-app/src/providers/App/index.js index 1a2d9a925..6304997eb 100644 --- a/packages/bruno-app/src/providers/App/index.js +++ b/packages/bruno-app/src/providers/App/index.js @@ -5,6 +5,7 @@ import { refreshScreenWidth } from 'providers/ReduxStore/slices/app'; import ConfirmAppClose from './ConfirmAppClose'; import useIpcEvents from './useIpcEvents'; import useTelemetry from './useTelemetry'; +import useParsedFileCacheIpc from './useParsedFileCacheIpc'; import StyledWrapper from './StyledWrapper'; import { version } from '../../../package.json'; @@ -13,6 +14,7 @@ export const AppContext = React.createContext(); export const AppProvider = (props) => { useTelemetry({ version }); useIpcEvents(); + useParsedFileCacheIpc(); const dispatch = useDispatch(); useEffect(() => { diff --git a/packages/bruno-app/src/providers/App/useIpcEvents.js b/packages/bruno-app/src/providers/App/useIpcEvents.js index 84b95e1a6..1f8e6fbe5 100644 --- a/packages/bruno-app/src/providers/App/useIpcEvents.js +++ b/packages/bruno-app/src/providers/App/useIpcEvents.js @@ -11,6 +11,7 @@ import { brunoConfigUpdateEvent, collectionAddDirectoryEvent, collectionAddFileEvent, + collectionBatchAddItems, collectionChangeFileEvent, collectionRenamedEvent, collectionUnlinkDirectoryEvent, @@ -101,6 +102,50 @@ const useIpcEvents = () => { } }; + // Batch handler for collection tree updates (performance optimization) + // Uses a single Redux dispatch to process all items, avoiding multiple re-renders + const _collectionTreeBatchUpdated = (batch) => { + if (!batch || !Array.isArray(batch) || batch.length === 0) { + return; + } + + if (window.__IS_DEV__) { + console.log('Batch update received:', batch.length, 'items'); + } + + // Separate batch items into those that can be bulk-processed vs those that need individual handling + const bulkItems = []; // addFile, addDir - can be processed in single reducer + const individualItems = []; // change, unlink, etc - need individual dispatches + + batch.forEach(({ eventType, payload }) => { + if (eventType === 'addDir' || eventType === 'addFile') { + bulkItems.push({ eventType, payload }); + } else { + individualItems.push({ eventType, payload }); + } + }); + + // Process bulk items in a single dispatch (addFile and addDir) + if (bulkItems.length > 0) { + dispatch(collectionBatchAddItems({ items: bulkItems })); + } + + // Process remaining items individually (these are typically rare during mount) + individualItems.forEach(({ eventType, payload }) => { + if (eventType === 'change') { + dispatch(collectionChangeFileEvent({ file: payload })); + } else if (eventType === 'unlink') { + dispatch(collectionUnlinkFileEvent({ file: payload })); + } else if (eventType === 'unlinkDir') { + dispatch(collectionUnlinkDirectoryEvent({ directory: payload })); + } else if (eventType === 'addEnvironmentFile') { + dispatch(collectionAddEnvFileEvent(payload)); + } else if (eventType === 'unlinkEnvironmentFile') { + dispatch(collectionUnlinkEnvFileEvent(payload)); + } + }); + }; + const _apiSpecTreeUpdated = (type, val) => { if (window.__IS_DEV__) { console.log('API Spec update:', type); @@ -118,6 +163,8 @@ const useIpcEvents = () => { const removeCollectionTreeUpdateListener = ipcRenderer.on('main:collection-tree-updated', _collectionTreeUpdated); + const removeCollectionTreeBatchUpdateListener = ipcRenderer.on('main:collection-tree-batch-updated', _collectionTreeBatchUpdated); + const removeApiSpecTreeUpdateListener = ipcRenderer.on('main:apispec-tree-updated', _apiSpecTreeUpdated); const removeOpenCollectionListener = ipcRenderer.on('main:collection-opened', (pathname, uid, brunoConfig) => { @@ -340,6 +387,7 @@ const useIpcEvents = () => { return () => { removeCollectionTreeUpdateListener(); + removeCollectionTreeBatchUpdateListener(); removeApiSpecTreeUpdateListener(); removeOpenCollectionListener(); removeOpenWorkspaceListener(); diff --git a/packages/bruno-app/src/providers/App/useParsedFileCacheIpc.js b/packages/bruno-app/src/providers/App/useParsedFileCacheIpc.js new file mode 100644 index 000000000..426bf8c14 --- /dev/null +++ b/packages/bruno-app/src/providers/App/useParsedFileCacheIpc.js @@ -0,0 +1,60 @@ +import { useEffect } from 'react'; +import { isElectron } from 'utils/common/platform'; +import { parsedFileCacheStore } from 'store/parsedFileCache'; + +const useParsedFileCacheIpc = () => { + useEffect(() => { + if (!isElectron()) { + return () => {}; + } + + const { ipcRenderer } = window; + + const handleCacheRequest = async (operation, requestId, ...args) => { + try { + let result = null; + switch (operation) { + case 'getEntry': + result = await parsedFileCacheStore.getEntry(...args); + break; + case 'setEntry': + await parsedFileCacheStore.setEntry(...args); + break; + case 'invalidate': + await parsedFileCacheStore.invalidate(...args); + break; + case 'invalidateCollection': + await parsedFileCacheStore.invalidateCollection(...args); + break; + case 'invalidateDirectory': + await parsedFileCacheStore.invalidateDirectory(...args); + break; + case 'getStats': + result = await parsedFileCacheStore.getStats(); + break; + case 'clear': + await parsedFileCacheStore.clear(); + break; + default: + throw new Error(`Unknown cache operation: ${operation}`); + } + ipcRenderer.send('renderer:parsed-file-cache-response', { requestId, success: true, data: result }); + } catch (error) { + ipcRenderer.send('renderer:parsed-file-cache-response', { requestId, success: false, error: error.message }); + } + }; + + const removeListener = ipcRenderer.on('main:parsed-file-cache-request', handleCacheRequest); + + // Prune old cache entries on startup + parsedFileCacheStore.prune().catch((err) => { + console.error('ParsedFileCacheStore: Error during startup prune:', err); + }); + + return () => { + removeListener(); + }; + }, []); +}; + +export default useParsedFileCacheIpc; diff --git a/packages/bruno-app/src/providers/ReduxStore/middlewares/tasks/middleware.js b/packages/bruno-app/src/providers/ReduxStore/middlewares/tasks/middleware.js index 3d0730e42..969dd8d62 100644 --- a/packages/bruno-app/src/providers/ReduxStore/middlewares/tasks/middleware.js +++ b/packages/bruno-app/src/providers/ReduxStore/middlewares/tasks/middleware.js @@ -4,7 +4,7 @@ import filter from 'lodash/filter'; import { createListenerMiddleware } from '@reduxjs/toolkit'; import { removeTaskFromQueue } from 'providers/ReduxStore/slices/app'; import { addTab } from 'providers/ReduxStore/slices/tabs'; -import { collectionAddFileEvent, collectionChangeFileEvent } from 'providers/ReduxStore/slices/collections'; +import { collectionAddFileEvent, collectionChangeFileEvent, collectionBatchAddItems } from 'providers/ReduxStore/slices/collections'; import { findCollectionByUid, findItemInCollectionByPathname, getDefaultRequestPaneTab, findItemInCollectionByItemUid } from 'utils/collections/index'; import { taskTypes } from './utils'; @@ -51,6 +51,57 @@ taskMiddleware.startListening({ } }); +/* + * When files are added via batch processing (e.g., during collection mount or when new files are created), + * we need to check if any of the added files match pending OPEN_REQUEST tasks. + * This handles the case where file additions go through the batch reducer instead of individual events. + */ +taskMiddleware.startListening({ + actionCreator: collectionBatchAddItems, + effect: (action, listenerApi) => { + const state = listenerApi.getState(); + const items = action.payload?.items || []; + + // Extract all addFile events from the batch + const addFileItems = items.filter((item) => item.eventType === 'addFile'); + if (addFileItems.length === 0) return; + + const openRequestTasks = filter(state.app.taskQueue, { type: taskTypes.OPEN_REQUEST }); + if (openRequestTasks.length === 0) return; + + each(addFileItems, ({ payload: file }) => { + const collectionUid = file?.meta?.collectionUid; + if (!collectionUid) return; + + each(openRequestTasks, (task) => { + if (collectionUid === task.collectionUid && file?.meta?.pathname === task.itemPathname) { + const collection = findCollectionByUid(state.collections.collections, collectionUid); + if (collection && collection.mountStatus === 'mounted' && !collection.isLoading) { + const item = findItemInCollectionByPathname(collection, task.itemPathname); + const isTransient = item?.isTransient ?? false; + if (item) { + listenerApi.dispatch( + addTab({ + uid: item.uid, + collectionUid: collection.uid, + requestPaneTab: getDefaultRequestPaneTab(item), + preview: !isTransient + }) + ); + } + } + + listenerApi.dispatch( + removeTaskFromQueue({ + taskUid: task.uid + }) + ); + } + }); + }); + } +}); + /* * When an example is created or cloned, a task to open the example is added to the queue. * We wait for the File IO to complete, after which the "collectionChangeFileEvent" gets dispatched. diff --git a/packages/bruno-app/src/providers/ReduxStore/slices/collections/index.js b/packages/bruno-app/src/providers/ReduxStore/slices/collections/index.js index dc7220c74..3f7317b36 100644 --- a/packages/bruno-app/src/providers/ReduxStore/slices/collections/index.js +++ b/packages/bruno-app/src/providers/ReduxStore/slices/collections/index.js @@ -162,6 +162,7 @@ export const collectionsSlice = createSlice({ collection.settingsSelectedTab = 'overview'; collection.folderLevelSettingsSelectedTab = {}; collection.allTags = []; // Initialize collection-level tags + collection.isLoading = false; // Collection mount status is used to track the mount status of the collection // values can be 'unmounted', 'mounting', 'mounted' @@ -2769,6 +2770,159 @@ export const collectionsSlice = createSlice({ addDepth(collection.items); } }, + // Batch reducer for adding multiple files/directories in a single state update + // This is a performance optimization to avoid multiple re-renders during collection mount + collectionBatchAddItems: (state, action) => { + const { items } = action.payload; + if (!items || !Array.isArray(items) || items.length === 0) { + return; + } + + // Group items by collection to minimize lookups + const itemsByCollection = new Map(); + for (const item of items) { + const collectionUid = item.payload?.meta?.collectionUid || item.payload?.collectionUid; + if (!collectionUid) continue; + + if (!itemsByCollection.has(collectionUid)) { + itemsByCollection.set(collectionUid, []); + } + itemsByCollection.get(collectionUid).push(item); + } + + // Process each collection's items + for (const [collectionUid, collectionItems] of itemsByCollection) { + const collection = findCollectionByUid(state.collections, collectionUid); + if (!collection) continue; + const tempDirectory = state.tempDirectories?.[collectionUid]; + + // Process directories first to ensure folder structure exists + const directories = collectionItems.filter((i) => i.eventType === 'addDir'); + const files = collectionItems.filter((i) => i.eventType === 'addFile'); + + // Add directories + for (const { payload: dir } of directories) { + const isTransientDir = tempDirectory && dir.meta.pathname.startsWith(tempDirectory); + const subDirectories = getSubdirectoriesFromRoot(collection.pathname, dir.meta.pathname); + let currentPath = collection.pathname; + let currentSubItems = collection.items; + for (const directoryName of subDirectories) { + let childItem = currentSubItems.find((f) => f.type === 'folder' && f.filename === directoryName); + currentPath = path.join(currentPath, directoryName); + if (!childItem) { + childItem = { + uid: dir?.meta?.uid || uuid(), + pathname: currentPath, + name: dir?.meta?.name || directoryName, + seq: dir?.meta?.seq, + filename: directoryName, + collapsed: true, + type: 'folder', + items: [], + isTransient: isTransientDir + }; + currentSubItems.push(childItem); + } else if (isTransientDir && !childItem.isTransient) { + childItem.isTransient = true; + } + currentSubItems = childItem.items; + } + } + + // Add files + for (const { payload: file } of files) { + const isCollectionRoot = file.meta.collectionRoot ? true : false; + const isFolderRoot = file.meta.folderRoot ? true : false; + const isTransientFile = tempDirectory && file.meta.pathname.startsWith(tempDirectory); + if (isCollectionRoot) { + collection.root = mergeRootWithPreservedUids(collection.root, file.data); + continue; + } + + if (isFolderRoot) { + const folderPath = path.dirname(file.meta.pathname); + const folderItem = findItemInCollectionByPathname(collection, folderPath); + if (folderItem) { + if (file?.data?.meta?.name) { + folderItem.name = file?.data?.meta?.name; + } + folderItem.root = mergeRootWithPreservedUids(folderItem.root, file.data); + if (file?.data?.meta?.seq) { + folderItem.seq = file.data?.meta?.seq; + } + } + continue; + } + + const dirname = path.dirname(file.meta.pathname); + const subDirectories = getSubdirectoriesFromRoot(collection.pathname, dirname); + let currentPath = collection.pathname; + let currentSubItems = collection.items; + for (const directoryName of subDirectories) { + let childItem = currentSubItems.find((f) => f.type === 'folder' && f.filename === directoryName); + currentPath = path.join(currentPath, directoryName); + if (!childItem) { + childItem = { + uid: uuid(), + pathname: currentPath, + name: directoryName, + collapsed: true, + type: 'folder', + items: [], + isTransient: isTransientFile + }; + currentSubItems.push(childItem); + } else if (isTransientFile && !childItem.isTransient) { + childItem.isTransient = true; + } + currentSubItems = childItem.items; + } + + if (file.meta.name !== 'folder.bru' && !currentSubItems.find((f) => f.name === file.meta.name)) { + const currentItem = find(currentSubItems, (i) => i.uid === file.data.uid); + if (currentItem) { + currentItem.name = file.data.name; + currentItem.type = file.data.type; + currentItem.seq = file.data.seq; + currentItem.tags = file.data.tags; + currentItem.request = mergeRequestWithPreservedUids(currentItem.request, file.data.request); + currentItem.filename = file.meta.name; + currentItem.pathname = file.meta.pathname; + currentItem.settings = file.data.settings; + currentItem.examples = file.data.examples; + currentItem.draft = null; + currentItem.partial = file.partial; + currentItem.loading = file.loading; + currentItem.size = file.size; + currentItem.error = file.error; + currentItem.isTransient = isTransientFile; + } else { + currentSubItems.push({ + uid: file.data.uid, + name: file.data.name, + type: file.data.type, + seq: file.data.seq, + tags: file.data.tags, + request: file.data.request, + settings: file.data.settings, + examples: file.data.examples, + filename: file.meta.name, + pathname: file.meta.pathname, + draft: null, + partial: file.partial, + loading: file.loading, + size: file.size, + error: file.error, + isTransient: isTransientFile + }); + } + } + } + + // Call addDepth once per collection after all items are added + addDepth(collection.items); + } + }, collectionChangeFileEvent: (state, action) => { const { file } = action.payload; const isCollectionRoot = file.meta.collectionRoot ? true : false; @@ -3662,6 +3816,7 @@ export const { updateCollectionProtobuf, collectionAddFileEvent, collectionAddDirectoryEvent, + collectionBatchAddItems, collectionChangeFileEvent, collectionUnlinkFileEvent, collectionUnlinkDirectoryEvent, diff --git a/packages/bruno-app/src/store/parsedFileCache.js b/packages/bruno-app/src/store/parsedFileCache.js new file mode 100644 index 000000000..416e3577c --- /dev/null +++ b/packages/bruno-app/src/store/parsedFileCache.js @@ -0,0 +1,249 @@ +import { openDB } from 'idb'; +import path from 'utils/common/path'; + +const DB_NAME = 'bruno-parsed-file-cache'; +const STORE_NAME = 'parsedFiles'; +const DB_VERSION = 1; +const CACHE_VERSION = '1.0.0'; +const DEFAULT_MAX_AGE_MS = 30 * 24 * 60 * 60 * 1000; // 30 days + +let dbPromise = null; + +const getDB = () => { + if (!dbPromise) { + dbPromise = openDB(DB_NAME, DB_VERSION, { + upgrade(db) { + if (!db.objectStoreNames.contains(STORE_NAME)) { + const store = db.createObjectStore(STORE_NAME, { keyPath: 'key' }); + store.createIndex('collectionPath', 'collectionPath'); + store.createIndex('parsedAt', 'parsedAt'); + } + } + }).catch((err) => { + dbPromise = null; + throw err; + }); + } + return dbPromise; +}; + +const generateKey = (collectionPath, filePath) => { + return `${collectionPath}↝${filePath}`; +}; + +export const parsedFileCacheStore = { + async getEntry(collectionPath, filePath) { + try { + const db = await getDB(); + const key = generateKey(collectionPath, filePath); + const entry = await db.get(STORE_NAME, key); + + if (entry && typeof entry.mtimeMs === 'number' && entry.parsedData) { + return { + mtimeMs: entry.mtimeMs, + parsedData: entry.parsedData + }; + } + } catch (error) { + console.error('ParsedFileCacheStore: Error reading cache entry:', error); + } + return null; + }, + + async setEntry(collectionPath, filePath, entry, retryAfterEviction = true) { + try { + const db = await getDB(); + const key = generateKey(collectionPath, filePath); + const cacheEntry = { + key, + collectionPath, + filePath, + mtimeMs: entry.mtimeMs, + parsedData: entry.parsedData, + parsedAt: Date.now() + }; + await db.put(STORE_NAME, cacheEntry); + } catch (error) { + // Handle QuotaExceededError by evicting old entries and retrying + const isQuotaError + = error.name === 'QuotaExceededError' + || error.code === 22 // Legacy Safari + || (error.code === 1014 && error.name === 'NS_ERROR_DOM_QUOTA_REACHED'); // Firefox + + if (isQuotaError && retryAfterEviction) { + console.warn('ParsedFileCacheStore: Quota exceeded, evicting old entries...'); + const evicted = await this.evictLRU(); + if (evicted > 0) { + // Retry the write after eviction + return this.setEntry(collectionPath, filePath, entry, false); + } + console.warn('ParsedFileCacheStore: No entries to evict, cache write skipped'); + } else { + console.error('ParsedFileCacheStore: Error writing cache entry:', error); + } + } + }, + + async invalidate(collectionPath, filePath) { + try { + const db = await getDB(); + const key = generateKey(collectionPath, filePath); + await db.delete(STORE_NAME, key); + } catch (error) { + console.error('ParsedFileCacheStore: Error invalidating cache entry:', error); + } + }, + + async invalidateCollection(collectionPath) { + try { + const db = await getDB(); + const tx = db.transaction(STORE_NAME, 'readwrite'); + const index = tx.store.index('collectionPath'); + + let cursor = await index.openCursor(IDBKeyRange.only(collectionPath)); + while (cursor) { + await cursor.delete(); + cursor = await cursor.continue(); + } + await tx.done; + } catch (error) { + console.error('ParsedFileCacheStore: Error invalidating collection cache:', error); + } + }, + + async invalidateDirectory(collectionPath, dirPath) { + try { + const db = await getDB(); + const tx = db.transaction(STORE_NAME, 'readwrite'); + const index = tx.store.index('collectionPath'); + const normalizedDirPath = dirPath.endsWith(path.sep) ? dirPath : `${dirPath}${path.sep}`; + + let cursor = await index.openCursor(IDBKeyRange.only(collectionPath)); + while (cursor) { + if (cursor.value.filePath.startsWith(normalizedDirPath)) { + await cursor.delete(); + } + cursor = await cursor.continue(); + } + await tx.done; + } catch (error) { + console.error('ParsedFileCacheStore: Error invalidating directory cache:', error); + } + }, + + async moveEntry(collectionPath, oldFilePath, newFilePath) { + try { + const entry = await this.getEntry(collectionPath, oldFilePath); + if (entry) { + await this.invalidate(collectionPath, oldFilePath); + await this.setEntry(collectionPath, newFilePath, { + mtimeMs: entry.mtimeMs, + parsedData: entry.parsedData + }); + } + } catch (error) { + console.error('ParsedFileCacheStore: Error moving cache entry:', error); + } + }, + + async prune(maxAgeMs = DEFAULT_MAX_AGE_MS) { + try { + const db = await getDB(); + const cutoff = Date.now() - maxAgeMs; + const tx = db.transaction(STORE_NAME, 'readwrite'); + const index = tx.store.index('parsedAt'); + + let cursor = await index.openCursor(IDBKeyRange.upperBound(cutoff)); + while (cursor) { + await cursor.delete(); + cursor = await cursor.continue(); + } + await tx.done; + } catch (error) { + console.error('ParsedFileCacheStore: Error pruning cache:', error); + } + }, + + /** + * Evict least recently used entries when quota is exceeded. + * Removes approximately 20% of the oldest entries to free up space. + * @returns {Promise} Number of entries evicted + */ + async evictLRU(percentageToEvict = 0.2) { + try { + const db = await getDB(); + const totalCount = await db.count(STORE_NAME); + + if (totalCount === 0) { + return 0; + } + + const countToEvict = Math.max(1, Math.floor(totalCount * percentageToEvict)); + + const tx = db.transaction(STORE_NAME, 'readwrite'); + const index = tx.store.index('parsedAt'); + + let cursor = await index.openCursor(); + let evicted = 0; + + while (cursor && evicted < countToEvict) { + await cursor.delete(); + evicted++; + cursor = await cursor.continue(); + } + + await tx.done; + console.log(`ParsedFileCacheStore: Evicted ${evicted} LRU entries to free up space`); + return evicted; + } catch (error) { + console.error('ParsedFileCacheStore: Error during LRU eviction:', error); + return 0; + } + }, + + async clear() { + try { + const db = await getDB(); + await db.clear(STORE_NAME); + } catch (error) { + console.error('ParsedFileCacheStore: Error clearing cache:', error); + } + }, + + async getStats() { + try { + const db = await getDB(); + + // Use count() for O(1) total files count + const totalFiles = await db.count(STORE_NAME); + + // Count unique collections using index with unique cursor + const tx = db.transaction(STORE_NAME, 'readonly'); + const index = tx.store.index('collectionPath'); + let totalCollections = 0; + + // Use openKeyCursor with 'nextunique' to count unique collection paths + let cursor = await index.openKeyCursor(null, 'nextunique'); + while (cursor) { + totalCollections++; + cursor = await cursor.continue(); + } + + return { + version: CACHE_VERSION, + totalCollections, + totalFiles + }; + } catch (error) { + console.error('ParsedFileCacheStore: Error getting stats:', error); + return { + version: CACHE_VERSION, + totalCollections: 0, + totalFiles: 0, + error: error.message + }; + } + } +}; + +export default parsedFileCacheStore; diff --git a/packages/bruno-electron/src/app/collection-tree-batcher.js b/packages/bruno-electron/src/app/collection-tree-batcher.js new file mode 100644 index 000000000..62bf1cbe3 --- /dev/null +++ b/packages/bruno-electron/src/app/collection-tree-batcher.js @@ -0,0 +1,196 @@ +/** + * CollectionTreeBatcher - Batches IPC events to reduce Redux dispatch overhead. + * + * Instead of sending individual 'main:collection-tree-updated' events for each file, + * this batcher collects events and sends them in batches, reducing the number of + * Redux updates and improving UI performance during collection mounting. + * + * Flush triggers: + * - Time-based: Every DISPATCH_INTERVAL_MS (200ms) + * - Size-based: When batch reaches MAX_BATCH_SIZE (300 items) + * - Manual: Call flush() directly (e.g., on watcher 'ready' event) + */ + +const DISPATCH_INTERVAL_MS = 200; +const MAX_BATCH_SIZE = 300; + +class CollectionTreeBatcher { + constructor(win, collectionUid) { + this.win = win; + this.queue = []; + this.timer = null; + this.isDestroyed = false; + // Bind methods + // We need to bind the methods because these are being called as callbacks to + // chokidar's add, addDir, change, unlink, unlinkDir events + + this.add = this.add.bind(this); + this.flush = this.flush.bind(this); + this._scheduleFlush = this._scheduleFlush.bind(this); + } + + /** + * Check if the window is still valid for sending events + */ + _isWindowValid() { + return this.win && !this.win.isDestroyed() && !this.isDestroyed; + } + + /** + * Schedule a flush after the dispatch interval + */ + _scheduleFlush() { + if (this.timer || !this._isWindowValid()) { + return; + } + + this.timer = setTimeout(() => { + this.timer = null; + this.flush(); + }, DISPATCH_INTERVAL_MS); + } + + /** + * Add an event to the batch queue + * @param {string} eventType - The event type ('addFile', 'addDir', 'change', 'unlink', 'unlinkDir') + * @param {object} payload - The event payload + */ + add(eventType, payload) { + if (!this._isWindowValid()) { + return; + } + + this.queue.push({ + eventType, + payload + }); + + // Flush immediately if batch is full + if (this.queue.length >= MAX_BATCH_SIZE) { + this.flush(); + } else { + this._scheduleFlush(); + } + } + + /** + * Flush the current batch to the renderer + */ + flush() { + // Clear any pending timer + if (this.timer) { + clearTimeout(this.timer); + this.timer = null; + } + + if (this.queue.length === 0 || !this._isWindowValid()) { + return; + } + + // Take all items from the queue + // This is a copy-type operation to avoid mutating the original + // Splice returns the deleted items + const batch = this.queue.splice(0); + + try { + // Send the batch to the renderer + this.win.webContents.send('main:collection-tree-batch-updated', batch); + } catch (error) { + console.error('CollectionTreeBatcher: Error sending batch:', error); + } + } + + /** + * Get the current queue size + * @returns {number} - The number of items in the queue + */ + size() { + return this.queue.length; + } + + /** + * Clear the queue without sending + */ + clear() { + if (this.timer) { + clearTimeout(this.timer); + this.timer = null; + } + this.queue = []; + } + + /** + * Mark this batcher as destroyed (e.g., when window closes) + */ + destroy() { + this.isDestroyed = true; + this.clear(); + this.win = null; + } +} + +// Store for managing batchers per collection +const batchers = new Map(); + +/** + * Get the batcher key for a window and collection UID + * @param {BrowserWindow} win - The Electron BrowserWindow + * @param {string} collectionUid - The collection UID + * @returns {string} - The batcher key + */ +const getBatcherKey = (win, collectionUid) => { + return `${win.id}-${collectionUid}`; +}; + +/** + * Get or create a CollectionTreeBatcher for a window + * @param {BrowserWindow} win - The Electron BrowserWindow + * @param {string} collectionUid - The collection UID + * @returns {CollectionTreeBatcher} - The batcher instance + */ +const getBatcher = (win, collectionUid) => { + const batcherKey = getBatcherKey(win, collectionUid); + + if (!batchers.has(batcherKey)) { + const batcher = new CollectionTreeBatcher(win, collectionUid); + + // Clean up when window is closed + win.once('closed', () => { + const b = batchers.get(batcherKey); + if (b) { + b.destroy(); + batchers.delete(batcherKey); + } + }); + + batchers.set(batcherKey, batcher); + } + + return batchers.get(batcherKey); +}; + +/** + * Remove a batcher for a window + * @param {BrowserWindow} win - The Electron BrowserWindow + * @param {string} collectionUid - The collection UID + */ +const removeBatcher = (win, collectionUid) => { + const batcherKey = getBatcherKey(win, collectionUid); + const batcher = batchers.get(batcherKey); + + if (batcher) { + batcher.destroy(); + batchers.delete(batcherKey); + } +}; + +// Export with backward-compatible aliases +module.exports = { + CollectionTreeBatcher, + getBatcher, + removeBatcher, + // Backward-compatible aliases + BatchAggregator: CollectionTreeBatcher, + getAggregator: getBatcher, + removeAggregator: removeBatcher +}; diff --git a/packages/bruno-electron/src/app/collection-watcher.js b/packages/bruno-electron/src/app/collection-watcher.js index 5f4c5fda2..d94c1e11d 100644 --- a/packages/bruno-electron/src/app/collection-watcher.js +++ b/packages/bruno-electron/src/app/collection-watcher.js @@ -1,5 +1,6 @@ const _ = require('lodash'); const fs = require('fs'); +const fsPromises = require('fs').promises; const path = require('path'); const chokidar = require('chokidar'); const { @@ -26,6 +27,8 @@ const UiStateSnapshot = require('../store/ui-state-snapshot'); const { parseFileMeta, hydrateRequestWithUuid } = require('../utils/collection'); const { parseLargeRequestWithRedaction } = require('../utils/parse'); const { transformBrunoConfigAfterRead } = require('../utils/transformBrunoConfig'); +const { parsedFileCacheStore } = require('../store/parsed-file-cache-idb'); +const { getBatcher } = require('./collection-tree-batcher'); const dotEnvWatcher = require('./dotenv-watcher'); const MAX_FILE_SIZE = 2.5 * 1024 * 1024; @@ -309,61 +312,71 @@ const add = async (win, pathname, collectionUid, collectionPath, useWorkerThread } }; - const fileStats = fs.statSync(pathname); - let content = fs.readFileSync(pathname, 'utf8'); + const batcher = getBatcher(win, collectionUid); - // If worker thread is not used, we can directly parse the file - if (!useWorkerThread) { - try { + try { + const fileStats = await fsPromises.stat(pathname); + + const cachedEntry = await parsedFileCacheStore.getEntry(collectionPath, pathname); + if (cachedEntry && cachedEntry.mtimeMs === fileStats.mtimeMs) { + // Cache hit + file.data = cachedEntry.parsedData; + file.partial = false; + file.loading = false; + file.size = sizeInMB(fileStats?.size); + hydrateRequestWithUuid(file.data, pathname); + batcher.add('addFile', file); + watcher.markFileAsProcessed(win, collectionUid, pathname); + return; + } + + // Cache miss + const content = await fsPromises.readFile(pathname, 'utf8'); + + if (!useWorkerThread) { file.data = await parseRequest(content, { format }); file.partial = false; file.loading = false; file.size = sizeInMB(fileStats?.size); hydrateRequestWithUuid(file.data, pathname); - win.webContents.send('main:collection-tree-updated', 'addFile', file); - } catch (error) { - console.error(error); - } finally { + batcher.add('addFile', file); + + await parsedFileCacheStore.setEntry(collectionPath, pathname, { + mtimeMs: fileStats.mtimeMs, + parsedData: file.data + }); watcher.markFileAsProcessed(win, collectionUid, pathname); + return; } - return; - } - - try { - // we need to send a partial file info to the UI - // so that the UI can display the file in the collection tree - file.data = { - name: path.basename(pathname), - type: 'http-request' - }; - - const metaJson = parseFileMeta(content, format); - file.data = metaJson; - file.partial = true; - file.loading = false; - file.size = sizeInMB(fileStats?.size); - hydrateRequestWithUuid(file.data, pathname); - win.webContents.send('main:collection-tree-updated', 'addFile', file); if (fileStats.size < MAX_FILE_SIZE) { - // This is to update the loading indicator in the UI - file.data = metaJson; - file.partial = false; - file.loading = true; - hydrateRequestWithUuid(file.data, pathname); - win.webContents.send('main:collection-tree-updated', 'addFile', file); - - // This is to update the file info in the UI file.data = await parseRequestViaWorker(content, { format, filename: pathname }); file.partial = false; file.loading = false; + file.size = sizeInMB(fileStats?.size); hydrateRequestWithUuid(file.data, pathname); - win.webContents.send('main:collection-tree-updated', 'addFile', file); + batcher.add('addFile', file); + + await parsedFileCacheStore.setEntry(collectionPath, pathname, { + mtimeMs: fileStats.mtimeMs, + parsedData: file.data + }); + } else { + const metaJson = parseFileMeta(content, format); + file.data = metaJson; + file.partial = true; + file.loading = false; + file.size = sizeInMB(fileStats?.size); + hydrateRequestWithUuid(file.data, pathname); + batcher.add('addFile', file); } + + watcher.markFileAsProcessed(win, collectionUid, pathname); } catch (error) { + console.error(`Error processing file ${pathname}:`, error); file.data = { name: path.basename(pathname), type: 'http-request' @@ -373,10 +386,8 @@ const add = async (win, pathname, collectionUid, collectionPath, useWorkerThread }; file.partial = true; file.loading = false; - file.size = sizeInMB(fileStats?.size); hydrateRequestWithUuid(file.data, pathname); - win.webContents.send('main:collection-tree-updated', 'addFile', file); - } finally { + batcher.add('addFile', file); watcher.markFileAsProcessed(win, collectionUid, pathname); } } @@ -396,15 +407,16 @@ const addDirectory = async (win, pathname, collectionUid, collectionPath) => { const folderFilePath = path.join(pathname, `folder.${format}`); try { - if (fs.existsSync(folderFilePath)) { - let folderFileContent = fs.readFileSync(folderFilePath, 'utf8'); - let folderData = await parseFolder(folderFileContent, { format }); - name = folderData?.meta?.name || name; - seq = folderData?.meta?.seq; - } + await fsPromises.access(folderFilePath); + const folderFileContent = await fsPromises.readFile(folderFilePath, 'utf8'); + const folderData = await parseFolder(folderFileContent, { format }); + name = folderData?.meta?.name || name; + seq = folderData?.meta?.seq; } catch (error) { - console.error(`Error occured while parsing folder.${format} file`); - console.error(error); + if (error.code !== 'ENOENT') { + console.error(`Error occurred while parsing folder.${format} file`); + console.error(error); + } } const directory = { @@ -417,7 +429,8 @@ const addDirectory = async (win, pathname, collectionUid, collectionPath) => { } }; - win.webContents.send('main:collection-tree-updated', 'addDir', directory); + const batcher = getBatcher(win, collectionUid); + batcher.add('addDir', directory); }; const change = async (win, pathname, collectionUid, collectionPath) => { @@ -524,6 +537,9 @@ const change = async (win, pathname, collectionUid, collectionPath) => { const format = getCollectionFormat(collectionPath); if (hasRequestExtension(pathname, format)) { + // Invalidate cache for this file since it changed + await parsedFileCacheStore.invalidate(collectionPath, pathname); + try { const file = { meta: { @@ -544,6 +560,14 @@ const change = async (win, pathname, collectionUid, collectionPath) => { file.size = sizeInMB(fileStats?.size); hydrateRequestWithUuid(file.data, pathname); + + // Update cache with new parsed data + await parsedFileCacheStore.setEntry(collectionPath, pathname, { + mtimeMs: fileStats.mtimeMs, + parsedData: file.data + }); + + // Change events are not batched - they need immediate feedback win.webContents.send('main:collection-tree-updated', 'change', file); } catch (err) { console.error(err); @@ -551,7 +575,7 @@ const change = async (win, pathname, collectionUid, collectionPath) => { } }; -const unlink = (win, pathname, collectionUid, collectionPath) => { +const unlink = async (win, pathname, collectionUid, collectionPath) => { console.log(`watcher unlink: ${pathname}`); if (isEnvironmentsFolder(pathname, collectionPath)) { @@ -560,6 +584,9 @@ const unlink = (win, pathname, collectionUid, collectionPath) => { const format = getCollectionFormat(collectionPath); if (hasRequestExtension(pathname, format)) { + // Invalidate cache for deleted file + await parsedFileCacheStore.invalidate(collectionPath, pathname); + const basename = path.basename(pathname); const dirname = path.dirname(pathname); @@ -585,6 +612,8 @@ const unlinkDir = async (win, pathname, collectionUid, collectionPath) => { return; } + await parsedFileCacheStore.invalidateDirectory(collectionPath, pathname); + const format = getCollectionFormat(collectionPath); const folderFilePath = path.join(pathname, `folder.${format}`); @@ -607,7 +636,9 @@ const unlinkDir = async (win, pathname, collectionUid, collectionPath) => { }; const onWatcherSetupComplete = (win, watchPath, collectionUid, watcher) => { - // Mark discovery as complete + const batcher = getBatcher(win, collectionUid); + batcher.flush(); + watcher.completeCollectionDiscovery(win, collectionUid); const UiStateSnapshotStore = new UiStateSnapshot(); diff --git a/packages/bruno-electron/src/index.js b/packages/bruno-electron/src/index.js index a3402a3e1..2c7129336 100644 --- a/packages/bruno-electron/src/index.js +++ b/packages/bruno-electron/src/index.js @@ -40,6 +40,7 @@ const registerNetworkIpc = require('./ipc/network'); const registerCollectionsIpc = require('./ipc/collection'); const registerFilesystemIpc = require('./ipc/filesystem'); const registerPreferencesIpc = require('./ipc/preferences'); +const { parsedFileCacheStore } = require('./store/parsed-file-cache-idb'); const registerSystemMonitorIpc = require('./ipc/system-monitor'); const registerWorkspaceIpc = require('./ipc/workspace'); const registerApiSpecIpc = require('./ipc/apiSpec'); @@ -440,6 +441,9 @@ app.on('ready', async () => { }); }); + // Initialize the parsed file cache IPC handlers + parsedFileCacheStore.initialize(mainWindow); + // register all ipc handlers registerNetworkIpc(mainWindow); registerGlobalEnvironmentsIpc(mainWindow, globalEnvironmentsManager); diff --git a/packages/bruno-electron/src/ipc/preferences.js b/packages/bruno-electron/src/ipc/preferences.js index f9d1b387f..1a2deb28c 100644 --- a/packages/bruno-electron/src/ipc/preferences.js +++ b/packages/bruno-electron/src/ipc/preferences.js @@ -2,6 +2,7 @@ const { ipcMain, nativeTheme } = require('electron'); const { getPreferences, savePreferences } = require('../store/preferences'); const { getGitVersion } = require('../utils/git'); const { globalEnvironmentsStore } = require('../store/global-environments'); +const { parsedFileCacheStore } = require('../store/parsed-file-cache-idb'); const { getCachedSystemProxy, refreshSystemProxy } = require('../store/system-proxy'); const registerPreferencesIpc = (mainWindow) => { @@ -43,6 +44,25 @@ const registerPreferencesIpc = (mainWindow) => { nativeTheme.themeSource = theme; }); + ipcMain.handle('renderer:get-cache-stats', async () => { + try { + return await parsedFileCacheStore.getStats(); + } catch (error) { + console.error('Error getting cache stats:', error); + return { error: error.message }; + } + }); + + ipcMain.handle('renderer:purge-cache', async () => { + try { + await parsedFileCacheStore.clear(); + return { success: true }; + } catch (error) { + console.error('Error purging cache:', error); + return { success: false, error: error.message }; + } + }); + ipcMain.handle('renderer:get-system-proxy-variables', async () => { // Return cached value (initialized at app startup) const cachedProxy = getCachedSystemProxy(); diff --git a/packages/bruno-electron/src/store/parsed-file-cache-idb.js b/packages/bruno-electron/src/store/parsed-file-cache-idb.js new file mode 100644 index 000000000..6a65109b8 --- /dev/null +++ b/packages/bruno-electron/src/store/parsed-file-cache-idb.js @@ -0,0 +1,157 @@ +const { ipcMain } = require('electron'); +const { v4: uuidv4 } = require('uuid'); + +// Pending requests waiting for renderer response +const pendingRequests = new Map(); + +// Timeout for IPC requests (5 seconds) +const REQUEST_TIMEOUT = 5000; + +// Store reference to main window +let mainWindow = null; + +// Initialize the IPC response handler +const initializeCacheIpc = (win) => { + mainWindow = win; + + ipcMain.on('renderer:parsed-file-cache-response', (event, response) => { + const { requestId, success, data, error } = response; + const pending = pendingRequests.get(requestId); + + if (pending) { + pendingRequests.delete(requestId); + clearTimeout(pending.timeout); + + if (success) { + pending.resolve(data); + } else { + pending.reject(new Error(error || 'Unknown error')); + } + } + }); +}; + +// Send a request to the renderer and wait for response +const sendCacheRequest = (operation, ...args) => { + return new Promise((resolve, reject) => { + if (!mainWindow || mainWindow.isDestroyed()) { + resolve(null); + return; + } + + const requestId = uuidv4(); + + const timeout = setTimeout(() => { + pendingRequests.delete(requestId); + resolve(null); + }, REQUEST_TIMEOUT); + + pendingRequests.set(requestId, { resolve, reject, timeout }); + + mainWindow.webContents.send('main:parsed-file-cache-request', operation, requestId, ...args); + }); +}; + +class ParsedFileCacheStore { + constructor() { + this.initialized = false; + } + + initialize(win) { + if (!this.initialized) { + initializeCacheIpc(win); + this.initialized = true; + } + } + + async getEntry(collectionPath, filePath) { + try { + return await sendCacheRequest('getEntry', collectionPath, filePath); + } catch (error) { + console.error('ParsedFileCacheStore: Error reading cache entry:', error); + return null; + } + } + + async setEntry(collectionPath, filePath, entry) { + try { + await sendCacheRequest('setEntry', collectionPath, filePath, entry); + } catch (error) { + console.error('ParsedFileCacheStore: Error writing cache entry:', error); + } + } + + async invalidate(collectionPath, filePath) { + try { + await sendCacheRequest('invalidate', collectionPath, filePath); + } catch (error) { + console.error('ParsedFileCacheStore: Error invalidating cache entry:', error); + } + } + + async invalidateCollection(collectionPath) { + try { + await sendCacheRequest('invalidateCollection', collectionPath); + } catch (error) { + console.error('ParsedFileCacheStore: Error invalidating collection cache:', error); + } + } + + async invalidateDirectory(collectionPath, dirPath) { + try { + await sendCacheRequest('invalidateDirectory', collectionPath, dirPath); + } catch (error) { + console.error('ParsedFileCacheStore: Error invalidating directory cache:', error); + } + } + + async moveEntry(collectionPath, oldFilePath, newFilePath) { + const entry = await this.getEntry(collectionPath, oldFilePath); + if (entry) { + await this.invalidate(collectionPath, oldFilePath); + await this.setEntry(collectionPath, newFilePath, { + mtimeMs: entry.mtimeMs, + parsedData: entry.parsedData + }); + } + } + + async getStats() { + try { + const stats = await sendCacheRequest('getStats'); + return stats || { + version: '1.0.0', + totalCollections: 0, + totalFiles: 0 + }; + } catch (error) { + console.error('ParsedFileCacheStore: Error getting stats:', error); + return { + version: '1.0.0', + totalCollections: 0, + totalFiles: 0, + error: error.message + }; + } + } + + async clear() { + try { + await sendCacheRequest('clear'); + } catch (error) { + console.error('ParsedFileCacheStore: Error clearing cache:', error); + } + } + + async close() { + // No-op for IndexedDB version (managed by browser) + } +} + +// Singleton instance +const parsedFileCacheStore = new ParsedFileCacheStore(); + +module.exports = { + parsedFileCacheStore, + ParsedFileCacheStore +}; diff --git a/packages/bruno-electron/tests/app/collection-tree-batcher.spec.js b/packages/bruno-electron/tests/app/collection-tree-batcher.spec.js new file mode 100644 index 000000000..4cbe66c11 --- /dev/null +++ b/packages/bruno-electron/tests/app/collection-tree-batcher.spec.js @@ -0,0 +1,312 @@ +const { CollectionTreeBatcher, getBatcher, removeBatcher } = require('../../src/app/collection-tree-batcher'); + +// Mock BrowserWindow +const createMockWindow = (id = 1) => { + const listeners = {}; + return { + id, + isDestroyed: jest.fn(() => false), + once: jest.fn((event, callback) => { + listeners[event] = callback; + }), + emit: (event) => { + if (listeners[event]) { + listeners[event](); + } + }, + webContents: { + send: jest.fn() + } + }; +}; + +describe('CollectionTreeBatcher', () => { + beforeEach(() => { + jest.useFakeTimers(); + }); + + afterEach(() => { + jest.useRealTimers(); + }); + + describe('constructor', () => { + it('should initialize with empty queue and no timer', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + expect(batcher.queue).toEqual([]); + expect(batcher.timer).toBeNull(); + expect(batcher.isDestroyed).toBe(false); + }); + }); + + describe('add()', () => { + it('should add events to the queue', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file.bru' }); + + expect(batcher.queue).toHaveLength(1); + expect(batcher.queue[0]).toEqual({ + eventType: 'addFile', + payload: { path: '/test/file.bru' } + }); + }); + + it('should schedule a flush after adding an event', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file.bru' }); + + expect(batcher.timer).not.toBeNull(); + }); + + it('should not add events if window is destroyed', () => { + const win = createMockWindow(); + win.isDestroyed.mockReturnValue(true); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file.bru' }); + + expect(batcher.queue).toHaveLength(0); + }); + + it('should not add events if batcher is destroyed', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + batcher.destroy(); + + batcher.add('addFile', { path: '/test/file.bru' }); + + expect(batcher.queue).toHaveLength(0); + }); + }); + + describe('flush()', () => { + it('should send batch to renderer and clear queue', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file1.bru' }); + batcher.add('addDir', { path: '/test/folder' }); + + batcher.flush(); + + expect(win.webContents.send).toHaveBeenCalledWith('main:collection-tree-batch-updated', [ + { eventType: 'addFile', payload: { path: '/test/file1.bru' } }, + { eventType: 'addDir', payload: { path: '/test/folder' } } + ]); + expect(batcher.queue).toHaveLength(0); + }); + + it('should not send if queue is empty', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.flush(); + + expect(win.webContents.send).not.toHaveBeenCalled(); + }); + + it('should clear pending timer on flush', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file.bru' }); + expect(batcher.timer).not.toBeNull(); + + batcher.flush(); + expect(batcher.timer).toBeNull(); + }); + + it('should not send if window is destroyed', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file.bru' }); + win.isDestroyed.mockReturnValue(true); + + batcher.flush(); + + expect(win.webContents.send).not.toHaveBeenCalled(); + }); + }); + + describe('time-based flush', () => { + it('should auto-flush after DISPATCH_INTERVAL_MS (200ms)', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file.bru' }); + + expect(win.webContents.send).not.toHaveBeenCalled(); + + jest.advanceTimersByTime(200); + + expect(win.webContents.send).toHaveBeenCalledWith('main:collection-tree-batch-updated', [ + { eventType: 'addFile', payload: { path: '/test/file.bru' } } + ]); + }); + + it('should not schedule multiple timers', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file1.bru' }); + const firstTimer = batcher.timer; + + batcher.add('addFile', { path: '/test/file2.bru' }); + + expect(batcher.timer).toBe(firstTimer); + }); + }); + + describe('size-based flush', () => { + it('should auto-flush when reaching MAX_BATCH_SIZE (300)', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + // Add 299 events - should not flush + for (let i = 0; i < 299; i++) { + batcher.add('addFile', { path: `/test/file${i}.bru` }); + } + expect(win.webContents.send).not.toHaveBeenCalled(); + expect(batcher.queue).toHaveLength(299); + + // Add 300th event - should trigger flush + batcher.add('addFile', { path: '/test/file299.bru' }); + + expect(win.webContents.send).toHaveBeenCalledTimes(1); + expect(batcher.queue).toHaveLength(0); + }); + }); + + describe('size()', () => { + it('should return current queue size', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + expect(batcher.size()).toBe(0); + + batcher.add('addFile', { path: '/test/file1.bru' }); + expect(batcher.size()).toBe(1); + + batcher.add('addFile', { path: '/test/file2.bru' }); + expect(batcher.size()).toBe(2); + }); + }); + + describe('clear()', () => { + it('should clear the queue without sending', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file.bru' }); + batcher.clear(); + + expect(batcher.queue).toHaveLength(0); + expect(batcher.timer).toBeNull(); + expect(win.webContents.send).not.toHaveBeenCalled(); + }); + }); + + describe('destroy()', () => { + it('should mark batcher as destroyed and clear queue', () => { + const win = createMockWindow(); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file.bru' }); + batcher.destroy(); + + expect(batcher.isDestroyed).toBe(true); + expect(batcher.queue).toHaveLength(0); + expect(batcher.win).toBeNull(); + }); + }); + + describe('error handling', () => { + it('should handle send errors gracefully', () => { + const win = createMockWindow(); + win.webContents.send.mockImplementation(() => { + throw new Error('Window closed'); + }); + const batcher = new CollectionTreeBatcher(win, 'collection-1'); + const consoleSpy = jest.spyOn(console, 'error').mockImplementation(() => {}); + + batcher.add('addFile', { path: '/test/file.bru' }); + batcher.flush(); + + expect(consoleSpy).toHaveBeenCalledWith('CollectionTreeBatcher: Error sending batch:', expect.any(Error)); + consoleSpy.mockRestore(); + }); + }); +}); + +describe('getBatcher / removeBatcher', () => { + beforeEach(() => { + jest.useFakeTimers(); + }); + + afterEach(() => { + jest.useRealTimers(); + }); + + it('should create and return a batcher for a window', () => { + const win = createMockWindow(100); + const batcher = getBatcher(win, 'collection-1'); + + expect(batcher).toBeInstanceOf(CollectionTreeBatcher); + }); + + it('should return the same batcher for the same window and collection', () => { + const win = createMockWindow(101); + const batcher1 = getBatcher(win, 'collection-1'); + const batcher2 = getBatcher(win, 'collection-1'); + + expect(batcher1).toBe(batcher2); + }); + + it('should return different batchers for different collections', () => { + const win = createMockWindow(102); + const batcher1 = getBatcher(win, 'collection-1'); + const batcher2 = getBatcher(win, 'collection-2'); + + expect(batcher1).not.toBe(batcher2); + }); + + it('should return different batchers for different windows', () => { + const win1 = createMockWindow(103); + const win2 = createMockWindow(104); + const batcher1 = getBatcher(win1, 'collection-1'); + const batcher2 = getBatcher(win2, 'collection-1'); + + expect(batcher1).not.toBe(batcher2); + }); + + it('should clean up batcher when window is closed', () => { + const win = createMockWindow(105); + const batcher = getBatcher(win, 'collection-1'); + + batcher.add('addFile', { path: '/test/file.bru' }); + + // Simulate window close + win.emit('closed'); + + expect(batcher.isDestroyed).toBe(true); + }); + + it('should remove batcher with removeBatcher', () => { + const win = createMockWindow(106); + const batcher = getBatcher(win, 'collection-1'); + + removeBatcher(win, 'collection-1'); + + expect(batcher.isDestroyed).toBe(true); + + // Getting batcher again should create a new one + const newBatcher = getBatcher(win, 'collection-1'); + expect(newBatcher).not.toBe(batcher); + }); +});