🐳
whale
  • Getting started
  • Setup
    • Connection configuration
    • Selective indexing
    • Git setup
  • Additional features
    • Running SQL queries
    • Query automation and alerting
    • Jinja2 templating
  • Customization
    • Custom UI/UX
    • All customizations
  • For Developers
    • ~/.whale file structure
    • Custom extraction
Powered by GitBook
On this page

Was this helpful?

  1. For Developers

Custom extraction

Previous~/.whale file structure

Last updated 4 years ago

Was this helpful?

Whale supports custom metadata scrapers through plain python scripts, as long as these satisfy two criteria:

  • A list of all tables should be generated in ~/.whale/manifests/manifest.txt.

  • Table "stubs" should be stored as markdown files in ~/.whale/metadata/.

To register a build script, simply run wh init and follow the build-script workflow, manually add details (see ).

This can be particularly useful for organizations where metadata is already collected and stored in an intermediary repository (e.g. Airbnb or Lyft), or if you want to get metadata from a source we do not support. For an example, see the databuilder/build_script.py file in the whale github repo.

Connection configuration