prepDATA/1.1 — Conversion, Transformation and Plotting of Basic Data Files

Type:package State:production Servers:vm5, vm4, vm3, vm2, vm1

About

The package has the fundamental commands to validate, filter, convert, and pre-process CSV files.

sDL structure

Follows the standard sDL structure:

//%SCOPE description text

#include "somefile"

BEGIN-CODE [ prepDATA/1.1 | {package options} ]
{package commands}
END-CODE

Commented lines, in the CODE body text, start with “##”.

Package options

The package has the following options format:

{options} *.csv

-b    -- break will be on
-d    -- drop bad lines
-ib   -- input file is in bzip format
-iz   -- input files is in zip format
-ob   -- output file will be in bzip format
-oz   -- output files will be in zip format
-w    -- warn will be on
root  -- the base folder is the sDL rootdir instead of the package "data" folder

Namespaces and commands

The namespaces available are: “core” (the sDL default), “type1” (type1 commands) and “type2” (type2 commands).

&core

• codesep
• col-function
• col-merge
• copy
• exit
• fileplat
• filter
• header
• lines-with
• lines-without
• load-state
• numsep
• save-state
• save2rDATA
• save2source
• sep
• stop
• with pandas

&type1

• col-select
• csv2sql
• csvclean
• csvcut
• csvformat
• csvgrep
• csvjoin
• csvjson
• csvsort
• csvstack
• cut-head
• cut-tail
• iconv
• in2csv
• sql2csv
• xlsx2csv

&type2

• cat
• col-show
• look
• stat

Folders

The package folder structure is the following:

sDLroot/
|-- prepDATA/
|-- |-- data/       -- input/output data folder
|-- |-- logs/       -- folder for specific logs
|-- sDL-jobs/       -- output logs of last runs

Examples

From sDL projects

People Involved

• Developer: Eugénio Rocha (homepage)
• Testers: Kelly Murillo (score: 4)

Change Log

Version 1.1

2016/02/25 - Several improvements and bug fixes
2016/02/21 - Implementation of "with pandas begin {...} endwith"

Version 1.0

2016/02/01 - Added package to sDL

(Documentation updated on 2019-01-05 23:20)