Merge pull request #2 from seanpedrick-case/dev

Updated README.md to latest gradio version

Merge pull request #1 from seanpedrick-case/dev

Fixes on representation model, visualisations, and embeddings in CPU mode. Package updates and optimisation for compatibility

Changed llama-cpp version for cpu

Corrected reference to sentence transformers dependency. Updated Dockerfile packages

Updated requirements

Fix on returning GPU tensors to main function after embedding with zeroGPU. Representation model put under ZeroGPU spaces

Removed explicit references to cuda in functions where spaces GPU are loaded

zeroGPU spaces duration can now be defined from environment variable

Added random seed to topic_core_funcs

Trying to load in cuda only within spaces environment to enable zero GPU space to run successfully

Updated package versions in requirements files

Adjusted requirements for max available for Huggingface python==3.10 platform

Test update main requirements file for huggingface compatibility

Re-added TruncatedSVD dependency to topic_core_funcs.py

Debugged reference to random_seed in vectorisation and reference to torch in representation_model.py

Importing space package near start of app now to avoid issue with cuda being initialised before

Llama-cpp-python in GPU mode doesn't seem to work well with Bertopic on Huggingface, so downgrading that to CPU version

Rearranged functions for embeddings creation to be compatible with zero GPU space. Updated packages.

Added and replaced relevant files to download in download_model.py to allow for app use on AWS

Updated Dockerfile with latest packages

Added example of how to run function from command line. Updated packages. Embedding model default now smaller and at fp16.

Improved initial clean options. Now has option to return embeddings only.

Corrected minor Dockerfile package version issue

App now retains original index following cleaning to allow for referring back to original data

Now installed dependencies into correct folder in Dockerfile

Finally managed to enforce cpu torch install in Dockerfile

Further optimised Dockerfile and requirements (smaller torch installation now hopefully)

Transferring across installed packages from build stage in Dockerfile

Changed Dockerfile to multi-stage build to further reduce size

Trying to make container image smaller through Dockerfile

Minor changes to reduce Dockerfile size

Updated download_model.py to download pytorch .bin file

Removed some requirements from Dockerfile for AWS deployment to reduce container size

Added NUMBA_CACHE_DIR to Docker environmental variables

Allowed for app running on AWS to use smaller embedding model and not to load representation LLM (due to size restrictions).

Dockerfile now installs models directly into user folder instead of moving from base folder

Updated Gradio version for spaces. Updated Dockerfile to enable Llama.cpp build with Cmake

Only aggregate topics not 'other', allowed for minimum sentence length, default max_topics now will auto aggregate topics. Added Cognito Auth functionality (boto3 with AWS).

Can split passages into sentences. Improved embedding, LLM representation models, improved zero shot capabilities

Updated packages. Improve hierarchy vis. Better models - mixedbread and phi3. Now option to split texts into sentences before modelling.

Minor cleaning, csv formatting changes

Reduce outliers now more efficient and relabels with correct vectoriser. Default topic labels now tidier. Hiearchical topics outputs more useful for joining to df afterwards. Switched low resource reduction algorithm to UMAP as default is not good.

Should now parse custom regex correctly. Will now wipe previously created embeddings if 'low resource mode' option switched.

Allowed for uploading custom regex for cleaning. Fixed calculate all probabilities, reduce outliers. Added text tree for hierarchical modelling.

Upgraded to Gradio 4.16.0. Guide for converting to exe added.

Hopefully now LLM download from hub should work

Note about LLM not working now successfully added!

Added note to say that LLM representation is not currently working on the HF website