Report Bug · Request Feature · Ask Question · Report security bug
urls_organizer is a Python program that helps to organize a list of URLs. It can read URLs from a file or add a single URL, removes duplicate URLs, and sorts them in natural order.
Additionally, the program can validate the URLs by sending a GET request and checking for the ones that are not available anymore. The program uses multiprocessing to speed up the validation process, and the results are saved in a separate file. The output URL list can be saved in a file specified by the user. The program takes command-line arguments for its inputs and parameters.
The purpose of the program is to assist users in managing their lists of URLs by keeping them organized and validating their accuracy.
Table of Contents
Here's a breakdown of the packages needed and their versions:
- poetry >= 1.7.1 (only for manual installation)
- natsort >= 8.4.0
- platformdirs >= 3.11.0
- pyfiglet >= 1.0.2
- requests >= 2.31.0
- rich >= 13.5.2
- rich-argparse-plus >= 0.3.1.4
- tqdm >= 4.66.1
- validators >= 0.22.0
Note
The software has been developed and tested using Python 3.10.10. The minimum required version to run the software is Python 3.6. Although the software may work with previous versions, it is not guaranteed.
urls_organizer can be installed by running one of the following commands in your terminal. You can install this via the command-line with either curl, wget or another similar tool.
| Method | Command |
|---|---|
| curl | sh -c "$(curl -fsSL https://raw.githubusercontent.com/yisuschrist/urls_organizer/main/install.sh)" |
| wget | sh -c "$(wget -O- https://raw.githubusercontent.com/yisuschrist/urls_organizer/main/install.sh)" |
| fetch | sh -c "$(fetch -o - https://raw.githubusercontent.com/yisuschrist/urls_organizer/main/install.sh)" |
urls_organizer can be installed easily as a PyPI package. Just run the following command:
pip3 install urls_organizerImportant
For best practices and to avoid potential conflicts with your global Python environment, it is strongly recommended to install this program within a virtual environment. Avoid using the --user option for global installations. We highly recommend using pipx for a safe and isolated installation experience. Therefore, the appropriate command to install urls_organizer would be:
pipx install urls_organizerIf you prefer to install the program manually, follow these steps:
Warning
This will install the version from the latest commit, not the latest release.
-
Download the latest version of urls_organizer from this repository:
git clone https://github.com/yisuschrist/urls_organizer cd urls_organizer -
Install the dependencies:
poetry install
-
Run the following commands to install urls_organizer in your
/usr/bin/directory:sudo chmod +x urls_organizer sudo cp urls_organizer /usr/bin/
To run the urls_organizer script, you can use the following command:
urls_organizer [OPTIONS]where [OPTIONS] are the command line options described below:
Content of urls.txt:
https://www.youtube.com
http://www.google.com
https://www.facebook.coma
https://www.youtube.comExecute the program with the following command:
urls_organizer -rf urls.txt -sf urls_organized.txt -w 10Content of urls_organized.txt:
https://google.com
https://youtube.comContent of invalid_urls.txt:
https://www.facebook.comaBefore you participate in our delightful community, please read the code of conduct.
I'm far from being an expert and suspect there are many ways to improve – if you have ideas on how to make the configuration easier to maintain (and faster), don't hesitate to fork and send pull requests!
We also need people to test out pull requests. So take a look through the open issues and help where you can.
See Contributing for more details.
Planing to add the following features:
- Add support for multiple input/output files
- Add a full documentation in Wiki section
- Add uninstall bash script
- Add a Changelog / Release Notes
urls_organizer is released under the GPL-3.0 License
Thanks to Marvin Wendt for the installation script template. Original code of the Instl project can be found here.
I made a few modifications to the original script to fit my needs (error control, fetching of repository release content and installation of required dependencies) because I found problems when running the autogenerated script.