Professional Documents
Culture Documents
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz
Scoop What is it ?
For data import / export to Hadoop Uses Map jobs from Map Reduce Supports incremental loads Written in Java Licensed by Apache Uses plugins for new types of data source
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz
Data sliced into partitions Mappers transfer data Data types determined via meta data Many data transfer formats supported
i.e. CSV, Avro Hive Hbase ( use --hive-import flag ) ( use hbase* flags )
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz
Scoop Interfaces
for scheduling
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz
Scoop Example
An example scoop command to
bin/sqoop-import --connect jdbc:mysql://<mysql host>:<msql port>/db3 \ -username <username> \ -password <password> \ --table <tableName> \ --hive-table <Hive tableName> \ --create-hive-table \ --hive-import \ --hive-home <hive path>
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz
Scoop Architecture
Scoop has moved from
Scoop1 to Scoop 2 Changed from client to server install Now has web and command line access Server now accesses Hive & Hbase Oozie uses REST API
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz
Contact Us
www.semtech-solutions.co.nz info@semtech-solutions.co.nz
We offer IT project consultancy We are happy to hear about your problems You can just pay for those hours that you need To solve your problems