Skip to content

mcanais/ISTransfere

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ISTransferido

Download the files of all your courses in a brief (long time). Redo it, and only the new ones will be added.

All new files will be in the desired organized files folder 📥

Uses Scrapy's library.

Configuration

  1. Prepare the virtual environment
source ./install.sh
  1. Create the file with your credentials
touch .env
echo IST_ID=istxxxxx >> .env
echo FENIX_PASSWORD=your_password >> .env
  1. Edit the configuration to your liking
vim config.toml

Running

Download the files + organize the downloads into the organized folder

# Run spider, RUN!
scrapy crawl fenix_spider && python src/filter_downloads.py

Extra notes

  • Please do not change the download delay to something that can create too many requests to the servers

About

Web Scraper for automated curricular file download from the Fenix platform of Instituto Superior Técnico

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages

  • Python 98.0%
  • Nix 1.4%
  • Shell 0.6%