first commit

This commit is contained in:
meninjar
2026-04-14 01:23:34 +00:00
commit edfaa886ff
443 changed files with 1245931 additions and 0 deletions
@@ -0,0 +1,277 @@
# Flexible Database Seeder
A flexible and extensible database seeder for Go applications with GORM support.
## Features
- 🎯 **Flexible Configuration**: Support for any table structure
- 📊 **CSV Import**: Import data from CSV files with customizable mapping
- 🔄 **Batch Processing**: Efficient bulk inserts with configurable batch sizes
- 🧪 **Dry Run Mode**: Preview seeding without actual database changes
-**Validation**: Validate CSV files and configurations before seeding
- 📝 **Column Mapping**: Map CSV columns to struct fields flexibly
- 🗂️ **Multiple Tables**: Support for all master data tables
- 🔍 **Progress Tracking**: Detailed logging and error reporting
## Quick Start
### Build the Seeder CLI
```bash
make seeder-build
```
### List Available Tables
```bash
make seeder-list
# or
./bin/seeder list
```
### Seed a Specific Table
```bash
make seeder-seed TABLE=province
# or
./bin/seeder seed province
```
### Seed All Tables
```bash
make seeder-seed-all
# or
./bin/seeder seed all
```
### Dry Run (Preview)
```bash
make seeder-dry-run TABLE=ethnic
# or
./bin/seeder dry-run ethnic
```
### Validate Configuration
```bash
make seeder-validate
# or
./bin/seeder validate all
```
## Advanced Usage
### Command Line Options
```bash
# Seed with custom batch size and delete existing data
./bin/seeder seed province -batch-size=200 -delete-before
# Seed with custom CSV file path
./bin/seeder seed province -csv-path=/path/to/custom/provinces.csv
# Seed with custom table name
./bin/seeder seed province -table-name=provinces_backup
# Dry run with specific options
./bin/seeder dry-run ethnic -batch-size=10
```
### Makefile Targets
```bash
# Advanced seeding with options
make seeder-advanced TABLE=province BATCH_SIZE=200 DELETE_BEFORE=1
# Build seeder
make seeder-build
# Clean seeder binary
make clean-seeder
```
## Configuration
### Table Configuration
Each table has a configuration that includes:
- **TableName**: Database table name
- **CSVFile**: Path to CSV file
- **ColumnMap**: Mapping between CSV headers and struct fields
- **DeleteBefore**: Whether to delete existing data before seeding
- **BatchSize**: Number of records to insert per batch
### Default Registry
The seeder comes with pre-configured tables:
| Table | CSV File | Description |
|-------|----------|-------------|
| province | provinces.csv | Province data |
| regency | regencies.csv | Regency/City data |
| district | districts.csv | District data |
| village | villages.csv | Village data |
| ethnic | ethnics.csv | Ethnic groups |
| language | languages.csv | Languages |
| installation | installations.csv | Hospital installations |
| unit | units.csv | Medical units |
| specialist | specialists.csv | Medical specialists |
| subspecialist | subspecialists.csv | Medical subspecialists |
### CSV File Format
CSV files should follow these conventions:
1. **Header Row**: First row must contain column names
2. **Column Names**: Use snake_case or PascalCase
3. **Data Types**: Automatic type conversion is supported
4. **Empty Values**: Empty cells are handled gracefully
Example CSV format:
```csv
Code,Name,Status
001,General Medicine,true
002,Cardiology,true
003,Neurology,true
```
## Extending the Seeder
### Adding New Tables
1. **Create Entity Struct** (if not exists):
```go
type MyEntity struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code string `gorm:"uniqueIndex;not null"`
Name string `gorm:"not null"`
Status bool `gorm:"default:true"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
```
2. **Register in DefaultRegistry**:
```go
registry.Register("mytable", TableConfig{
TableName: "MyTable",
CSVFile: filepath.Join(basePath, "mytable.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"Name": "Name",
"Status": "Status",
},
DeleteBefore: true,
BatchSize: 50,
})
```
3. **Add to GetEntityByTableName**:
```go
case "mytable":
return &MyEntity{}
```
### Custom Column Mapping
The seeder supports flexible column mapping:
```go
ColumnMap: map[string]string{
"CSV_Column_Name": "StructFieldName",
"province_code": "ProvinceCode",
"full_name": "Name",
}
```
### Custom Type Conversion
For complex types, you can extend the `setFieldValue` function in `MasterSeeder`.
## Error Handling
The seeder provides detailed error messages for:
- **File Not Found**: CSV file doesn't exist
- **Invalid CSV Format**: Malformed CSV files
- **Database Errors**: Connection issues, constraint violations
- **Type Conversion**: Invalid data types
- **Validation**: Missing required fields
## Logging
The seeder provides comprehensive logging:
```
2024-01-01 12:00:00 [INFO] Starting seed for table: province
2024-01-01 12:00:00 [INFO] CSV file: internal/infrastructure/database/csv/provinces.csv
2024-01-01 12:00:00 [INFO] Batch size: 50
2024-01-01 12:00:00 [INFO] Delete before: true
2024-01-01 12:00:00 [INFO] Seeded province: 11 - ACEH
2024-01-01 12:00:00 [INFO] Seeding completed. 38 records processed from Province
```
## Performance Tips
1. **Batch Size**: Use larger batch sizes (100-500) for better performance
2. **Indexing**: Consider dropping indexes during seeding for large datasets
3. **Transactions**: Each batch is processed in a transaction
4. **Dry Run**: Always use dry-run first to validate data
5. **Delete Strategy**: Use `delete-before` for clean slate seeding
## Environment Variables
```bash
# Database connection (optional, defaults to localhost)
export DATABASE_URL="host=localhost user=postgres password=postgres dbname=health port=5432 sslmode=disable"
```
## Troubleshooting
### Common Issues
1. **CSV File Not Found**
```
Error: CSV file not found: internal/infrastructure/database/csv/provinces.csv
```
Solution: Ensure CSV files are in the correct directory
2. **Database Connection Failed**
```
Error: Failed to connect to database: ...
```
Solution: Check DATABASE_URL environment variable
3. **Invalid Column Mapping**
```
Error: Failed to set field X: field not found
```
Solution: Check ColumnMap configuration
4. **Type Conversion Error**
```
Error: Failed to parse int from 'ABC': ...
```
Solution: Check CSV data types match expected format
### Debug Mode
Use dry-run mode to preview data without inserting:
```bash
./bin/seeder dry-run province
```
## Contributing
1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Submit a pull request
## License
This seeder is part of the service-general project and follows the same license terms.
@@ -0,0 +1,399 @@
package seeders
import (
"encoding/csv"
"fmt"
"log"
"os"
"strings"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
// SeederCLI adalah command-line interface untuk seeder
type SeederCLI struct {
db *gorm.DB
registry *SeederRegistry
}
// NewSeederCLI membuat CLI baru
func NewSeederCLI(db *gorm.DB) *SeederCLI {
return &SeederCLI{
db: db,
registry: DefaultRegistry(),
}
}
// Run menjalankan CLI seeder
func (c *SeederCLI) Run(args []string) error {
if len(args) == 0 {
return c.showHelp()
}
command := args[0]
switch command {
case "list":
return c.listTables()
case "seed":
if len(args) < 2 {
return fmt.Errorf("usage: seeder seed <table-name|all> [options]")
}
return c.seedTable(args[1], args[2:])
case "dry-run":
if len(args) < 2 {
return fmt.Errorf("usage: seeder dry-run <table-name|all> [options]")
}
return c.dryRunTable(args[1], args[2:])
case "validate":
if len(args) < 2 {
return fmt.Errorf("usage: seeder validate <table-name|all>")
}
return c.validateTable(args[1])
default:
return fmt.Errorf("unknown command: %s", command)
}
}
// showHelp menampilkan bantuan
func (c *SeederCLI) showHelp() error {
help := `
Seeder CLI - Flexible Database Seeder
Usage:
seeder <command> [arguments]
Commands:
list Show all available tables
seed <table|all> Seed data for specific table or all tables
dry-run <table|all> Preview seeding without actual insertion
validate <table|all> Validate CSV files and configuration
Examples:
seeder list
seeder seed province
seeder seed all
seeder dry-run ethnic
seeder validate regency
Options:
-batch-size=N Set batch size for insertion (default: from config)
-delete-before Delete existing data before seeding
-no-delete Don't delete existing data (default)
-csv-path=PATH Override CSV file path
-table-name=NAME Override table name
Notes:
- CSV files should be in internal/infrastructure/database/csv/
- First row of CSV should be header with column names
- Column names in CSV will be mapped to struct fields
`
fmt.Println(help)
return nil
}
// listTables menampilkan daftar tabel yang tersedia
func (c *SeederCLI) listTables() error {
tables := c.registry.List()
fmt.Println("Available tables for seeding:")
fmt.Println(strings.Repeat("-", 50))
for _, table := range tables {
config, _ := c.registry.Get(table)
fmt.Printf("%-20s %s\n", table, config.CSVFile)
}
fmt.Printf("\nTotal: %d tables\n", len(tables))
return nil
}
// seedTable melakukan seeding untuk tabel tertentu
func (c *SeederCLI) seedTable(tableName string, options []string) error {
if tableName == "all" {
return c.seedAllTables(options)
}
config, exists := c.registry.Get(tableName)
if !exists {
return fmt.Errorf("table '%s' not found in registry", tableName)
}
// Parse options
opts := c.parseOptions(options)
// Override config dengan options
if opts.CSVPath != "" {
config.CSVFile = opts.CSVPath
}
if opts.TableName != "" {
config.TableName = opts.TableName
}
if opts.BatchSize > 0 {
config.BatchSize = opts.BatchSize
}
if opts.DeleteBefore {
config.DeleteBefore = true
}
// Validasi config
if err := config.Validate(); err != nil {
return fmt.Errorf("validation failed: %w", err)
}
// Get entity
entity := GetEntityByTableName(config.TableName)
if entity == nil {
return fmt.Errorf("entity not found for table '%s'", config.TableName)
}
// Create seeder
seederConfig := SeederConfig{
CSVPath: config.CSVFile,
TableName: config.TableName,
ColumnMap: config.ColumnMap,
SkipHeader: true,
BatchSize: config.BatchSize,
DeleteBefore: config.DeleteBefore,
DryRun: false,
}
seeder := NewMasterSeeder(c.db, seederConfig)
log.Printf("Starting seed for table: %s", config.TableName)
log.Printf("CSV file: %s", config.CSVFile)
log.Printf("Batch size: %d", config.BatchSize)
log.Printf("Delete before: %v", config.DeleteBefore)
if err := seeder.SeedFromCSV(entity); err != nil {
return fmt.Errorf("seeding failed: %w", err)
}
log.Printf("Successfully seeded table: %s", config.TableName)
return nil
}
// dryRunTable melakukan dry-run untuk tabel tertentu
func (c *SeederCLI) dryRunTable(tableName string, options []string) error {
if tableName == "all" {
return c.dryRunAllTables(options)
}
config, exists := c.registry.Get(tableName)
if !exists {
return fmt.Errorf("table '%s' not found in registry", tableName)
}
// Parse options
opts := c.parseOptions(options)
// Override config dengan options
if opts.CSVPath != "" {
config.CSVFile = opts.CSVPath
}
if opts.TableName != "" {
config.TableName = opts.TableName
}
// Validasi config
if err := config.Validate(); err != nil {
return fmt.Errorf("validation failed: %w", err)
}
// Get entity
entity := GetEntityByTableName(config.TableName)
if entity == nil {
return fmt.Errorf("entity not found for table '%s'", config.TableName)
}
// Create seeder dengan dry-run mode
seederConfig := SeederConfig{
CSVPath: config.CSVFile,
TableName: config.TableName,
ColumnMap: config.ColumnMap,
SkipHeader: true,
BatchSize: 10, // Smaller batch for dry-run
DeleteBefore: false,
DryRun: true,
}
seeder := NewMasterSeeder(c.db, seederConfig)
log.Printf("Starting dry-run for table: %s", config.TableName)
log.Printf("CSV file: %s", config.CSVFile)
if err := seeder.SeedFromCSV(entity); err != nil {
return fmt.Errorf("dry-run failed: %w", err)
}
log.Printf("Dry-run completed for table: %s", config.TableName)
return nil
}
// validateTable memvalidasi konfigurasi dan CSV file
func (c *SeederCLI) validateTable(tableName string) error {
if tableName == "all" {
return c.validateAllTables()
}
config, exists := c.registry.Get(tableName)
if !exists {
return fmt.Errorf("table '%s' not found in registry", tableName)
}
// Validasi config
if err := config.Validate(); err != nil {
return fmt.Errorf("validation failed: %w", err)
}
// Get entity
entity := GetEntityByTableName(config.TableName)
if entity == nil {
return fmt.Errorf("entity not found for table '%s'", config.TableName)
}
// Baca CSV file dan validasi
file, err := os.Open(config.CSVFile)
if err != nil {
return fmt.Errorf("failed to open CSV file: %w", err)
}
defer file.Close()
reader := csv.NewReader(file)
headers, err := reader.Read()
if err != nil {
return fmt.Errorf("failed to read CSV header: %w", err)
}
// Hitung jumlah baris
rowCount := 0
for {
_, err := reader.Read()
if err != nil {
break
}
rowCount++
}
fmt.Printf("Validation passed for table: %s\n", tableName)
fmt.Printf("CSV file: %s\n", config.CSVFile)
fmt.Printf("Headers: %v\n", headers)
fmt.Printf("Expected rows: %d\n", rowCount)
fmt.Printf("Column mapping: %v\n", config.ColumnMap)
fmt.Printf("Entity type: %T\n", entity)
return nil
}
// seedAllTables melakukan seeding untuk semua tabel
func (c *SeederCLI) seedAllTables(options []string) error {
tables := c.registry.List()
log.Printf("Starting seed for all tables (%d tables)", len(tables))
for _, table := range tables {
if err := c.seedTable(table, options); err != nil {
log.Printf("Error seeding table %s: %v", table, err)
continue
}
}
log.Printf("Completed seeding all tables")
return nil
}
// dryRunAllTables melakukan dry-run untuk semua tabel
func (c *SeederCLI) dryRunAllTables(options []string) error {
tables := c.registry.List()
log.Printf("Starting dry-run for all tables (%d tables)", len(tables))
for _, table := range tables {
if err := c.dryRunTable(table, options); err != nil {
log.Printf("Error in dry-run for table %s: %v", table, err)
continue
}
}
log.Printf("Completed dry-run for all tables")
return nil
}
// validateAllTables memvalidasi semua tabel
func (c *SeederCLI) validateAllTables() error {
tables := c.registry.List()
fmt.Printf("Validating all tables (%d tables):\n", len(tables))
fmt.Println(strings.Repeat("=", 60))
for _, table := range tables {
fmt.Printf("\n[%s]\n", table)
if err := c.validateTable(table); err != nil {
fmt.Printf("❌ Validation failed: %v\n", err)
} else {
fmt.Printf("✅ Validation passed\n")
}
}
return nil
}
// SeederOptions berisi parsed options
type SeederOptions struct {
CSVPath string
TableName string
BatchSize int
DeleteBefore bool
DryRun bool
}
// parseOptions mem-parsing command line options
func (c *SeederCLI) parseOptions(options []string) SeederOptions {
var opts SeederOptions
for _, option := range options {
switch {
case strings.HasPrefix(option, "-batch-size="):
fmt.Sscanf(option, "-batch-size=%d", &opts.BatchSize)
case option == "-delete-before":
opts.DeleteBefore = true
case strings.HasPrefix(option, "-csv-path="):
opts.CSVPath = strings.TrimPrefix(option, "-csv-path=")
case strings.HasPrefix(option, "-table-name="):
opts.TableName = strings.TrimPrefix(option, "-table-name=")
}
}
return opts
}
// MainCLI adalah entry point utama untuk CLI
func MainCLI() {
// Setup database connection
db, err := setupDatabase()
if err != nil {
log.Fatalf("Failed to connect to database: %v", err)
}
cli := NewSeederCLI(db)
if err := cli.Run(os.Args[1:]); err != nil {
log.Fatalf("Error: %v", err)
}
}
// setupDatabase setup koneksi database
func setupDatabase() (*gorm.DB, error) {
// Bisa diambil dari config atau environment variable
dsn := os.Getenv("DATABASE_URL")
if dsn == "" {
dsn = "host=localhost user=postgres password=postgres dbname=health port=5432 sslmode=disable"
}
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
return nil, err
}
return db, nil
}
@@ -0,0 +1,356 @@
package seeders
import (
"fmt"
"os"
"path/filepath"
"strings"
"time"
)
// TableConfig berisi konfigurasi untuk setiap tabel
type TableConfig struct {
TableName string
CSVFile string
Entity interface{}
ColumnMap map[string]string
DeleteBefore bool
BatchSize int
}
// SeederRegistry berisi registry untuk semua tabel yang bisa di-seed
type SeederRegistry struct {
tables map[string]TableConfig
}
// NewSeederRegistry membuat registry baru
func NewSeederRegistry() *SeederRegistry {
return &SeederRegistry{
tables: make(map[string]TableConfig),
}
}
// Register mendaftarkan tabel ke registry
func (r *SeederRegistry) Register(name string, config TableConfig) {
r.tables[name] = config
}
// Get mendapatkan konfigurasi tabel berdasarkan nama
func (r *SeederRegistry) Get(name string) (TableConfig, bool) {
config, exists := r.tables[name]
return config, exists
}
// List mengembalikan daftar nama tabel yang terdaftar
func (r *SeederRegistry) List() []string {
names := make([]string, 0, len(r.tables))
for name := range r.tables {
names = append(names, name)
}
return names
}
// DefaultRegistry berisi konfigurasi default untuk semua tabel
func DefaultRegistry() *SeederRegistry {
registry := NewSeederRegistry()
basePath := "internal/infrastructure/database/csv"
// Register Province
registry.Register("province", TableConfig{
TableName: "Province",
CSVFile: filepath.Join(basePath, "provinces.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"Name": "Name",
},
DeleteBefore: true,
BatchSize: 50,
})
// Register Regency
registry.Register("regency", TableConfig{
TableName: "Regency",
CSVFile: filepath.Join(basePath, "regencies.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"Province_Code": "ProvinceCode",
"Name": "Name",
},
DeleteBefore: true,
BatchSize: 100,
})
// Register District
registry.Register("district", TableConfig{
TableName: "District",
CSVFile: filepath.Join(basePath, "districts.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"Regency_Code": "RegencyCode",
"Name": "Name",
},
DeleteBefore: true,
BatchSize: 200,
})
// Register Village
registry.Register("village", TableConfig{
TableName: "Village",
CSVFile: filepath.Join(basePath, "villages.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"District_Code": "DistrictCode",
"Name": "Name",
},
DeleteBefore: true,
BatchSize: 500,
})
// Register Ethnic
registry.Register("ethnic", TableConfig{
TableName: "Ethnic",
CSVFile: filepath.Join(basePath, "ethnics.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"Name": "Name",
"Status": "Status",
},
DeleteBefore: true,
BatchSize: 50,
})
// Register Language
registry.Register("language", TableConfig{
TableName: "Language",
CSVFile: filepath.Join(basePath, "languages.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"Name": "Name",
"Status": "Status",
},
DeleteBefore: true,
BatchSize: 50,
})
// Register Installation
registry.Register("installation", TableConfig{
TableName: "Installation",
CSVFile: filepath.Join(basePath, "installations.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"Name": "Name",
"Status": "Status",
},
DeleteBefore: true,
BatchSize: 50,
})
// Register Unit
registry.Register("unit", TableConfig{
TableName: "Unit",
CSVFile: filepath.Join(basePath, "units.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"Name": "Name",
"Status": "Status",
},
DeleteBefore: true,
BatchSize: 50,
})
// Register Specialist
registry.Register("specialist", TableConfig{
TableName: "Specialist",
CSVFile: filepath.Join(basePath, "specialists.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"Name": "Name",
"Status": "Status",
},
DeleteBefore: true,
BatchSize: 50,
})
// Register SubSpecialist
registry.Register("subspecialist", TableConfig{
TableName: "SubSpecialist",
CSVFile: filepath.Join(basePath, "subspecialists.csv"),
ColumnMap: map[string]string{
"Code": "Code",
"Specialist_Code": "SpecialistCode",
"Name": "Name",
"Status": "Status",
},
DeleteBefore: true,
BatchSize: 50,
})
// Register RolPages (Menu Structure)
registry.Register("rol_pages", TableConfig{
TableName: "rol_pages",
Entity: &RolPages{},
DeleteBefore: true,
BatchSize: 20,
})
return registry
}
// ValidateConfig memvalidasi konfigurasi
func (c *TableConfig) Validate() error {
if c.TableName == "" {
return fmt.Errorf("table name is required")
}
if c.CSVFile == "" {
return fmt.Errorf("CSV file path is required")
}
// Cek apakah file CSV ada
if _, err := os.Stat(c.CSVFile); os.IsNotExist(err) {
return fmt.Errorf("CSV file not found: %s", c.CSVFile)
}
if c.BatchSize <= 0 {
c.BatchSize = 100
}
return nil
}
// GetEntityByTableName mendapatkan entity berdasarkan nama tabel
func GetEntityByTableName(tableName string) interface{} {
switch strings.ToLower(tableName) {
case "province":
return &Province{}
case "regency":
return &Regency{}
case "district":
return &District{}
case "village":
return &Village{}
// case "ethnic":
// return &Ethnic{}
case "language":
return &Language{}
case "installation":
return &Installation{}
case "unit":
return &Unit{}
case "specialist":
return &Specialist{}
case "subspecialist":
return &SubSpecialist{}
case "rol_pages":
return &RolPages{}
default:
return nil
}
}
// Entity definitions
type Province struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code string `gorm:"uniqueIndex;not null"`
Name string `gorm:"not null"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
type Regency struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code string `gorm:"uniqueIndex;not null"`
ProvinceCode string `gorm:"index;not null"`
Name string `gorm:"not null"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
type District struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code string `gorm:"uniqueIndex;not null"`
RegencyCode string `gorm:"index;not null"`
Name string `gorm:"not null"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
type Village struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code string `gorm:"uniqueIndex;not null"`
DistrictCode string `gorm:"index;not null"`
Name string `gorm:"not null"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
type Language struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code interface{} `gorm:"uniqueIndex;not null"`
Name interface{} `gorm:"not null"`
Status interface{} `gorm:"default:true"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
type Installation struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code interface{} `gorm:"uniqueIndex;not null"`
Name interface{} `gorm:"not null"`
Status interface{} `gorm:"default:true"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
type Unit struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code interface{} `gorm:"uniqueIndex;not null"`
Name interface{} `gorm:"not null"`
Status interface{} `gorm:"default:true"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
type Specialist struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code interface{} `gorm:"uniqueIndex;not null"`
Name interface{} `gorm:"not null"`
Status interface{} `gorm:"default:true"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
type SubSpecialist struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code interface{} `gorm:"uniqueIndex;not null"`
SpecialistCode interface{} `gorm:"index;not null"`
Name interface{} `gorm:"not null"`
Status interface{} `gorm:"default:true"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
// RolPages struct untuk menu hierarchy
type RolPages struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Name string `gorm:"not null;size:20"`
Icon string `gorm:"size:100"`
URL string `gorm:"not null"`
Level int16 `gorm:"not null;default:0"`
Sort int16 `gorm:"not null;default:0"`
Parent *int64 `gorm:"index"`
Active bool `gorm:"not null;default:true"`
CreatedAt time.Time `gorm:"not null;default:CURRENT_TIMESTAMP"`
UpdatedAt *time.Time
DeletedAt *time.Time `gorm:"index"`
}
@@ -0,0 +1,372 @@
package seeders
import (
"encoding/csv"
"fmt"
"log"
"os"
"reflect"
"strings"
"time"
"gorm.io/gorm"
)
// SeederConfig berisi konfigurasi untuk seeding
type SeederConfig struct {
CSVPath string
TableName string
ColumnMap map[string]string // mapping CSV header ke field struct
SkipHeader bool
BatchSize int
DryRun bool
DeleteBefore bool
}
// MasterSeeder adalah struct utama untuk seeding fleksibel
type MasterSeeder struct {
db *gorm.DB
config SeederConfig
}
// NewMasterSeeder membuat instance seeder baru
func NewMasterSeeder(db *gorm.DB, config SeederConfig) *MasterSeeder {
if config.BatchSize == 0 {
config.BatchSize = 100
}
return &MasterSeeder{
db: db,
config: config,
}
}
// SeedFromCSV melakukan seeding dari file CSV
func (s *MasterSeeder) SeedFromCSV(model interface{}) error {
if s.config.CSVPath == "" {
return fmt.Errorf("CSV path is required")
}
file, err := os.Open(s.config.CSVPath)
if err != nil {
return fmt.Errorf("failed to open CSV file: %w", err)
}
defer file.Close()
reader := csv.NewReader(file)
reader.Comma = ','
// Baca header
headers, err := reader.Read()
if err != nil {
return fmt.Errorf("failed to read CSV header: %w", err)
}
log.Printf("CSV Headers: %v", headers)
// Hapus data lama jika diminta
if s.config.DeleteBefore {
if err := s.db.Where("1=1").Delete(model).Error; err != nil {
return fmt.Errorf("failed to delete existing data: %w", err)
}
log.Printf("Deleted existing data from %s", s.config.TableName)
}
count := 0
batch := []interface{}{}
for {
record, err := reader.Read()
if err != nil {
if err.Error() == "EOF" {
break
}
log.Printf("Error reading row %d: %v", count+1, err)
continue
}
if len(record) < len(headers) {
log.Printf("Skipping incomplete row %d", count+1)
continue
}
// Buat instance model baru
instance := reflect.New(reflect.TypeOf(model).Elem()).Interface()
// Mapping data dari CSV ke struct
if err := s.mapCSVToStruct(headers, record, instance); err != nil {
log.Printf("Error mapping row %d: %v", count+1, err)
continue
}
// Set audit fields
s.setAuditFields(instance)
if s.config.DryRun {
log.Printf("Dry run - Would insert: %+v", instance)
} else {
batch = append(batch, instance)
// Insert batch jika sudah penuh
if len(batch) >= s.config.BatchSize {
if err := s.insertBatch(batch); err != nil {
return fmt.Errorf("failed to insert batch: %w", err)
}
count += len(batch)
batch = []interface{}{}
}
}
}
// Insert sisa batch
if len(batch) > 0 && !s.config.DryRun {
if err := s.insertBatch(batch); err != nil {
return fmt.Errorf("failed to insert final batch: %w", err)
}
count += len(batch)
}
log.Printf("Seeding completed. %d records processed from %s", count, s.config.TableName)
return nil
}
// mapCSVToStruct mapping data CSV ke struct berdasarkan column map atau reflection
func (s *MasterSeeder) mapCSVToStruct(headers []string, record []string, instance interface{}) error {
v := reflect.ValueOf(instance).Elem()
t := v.Type()
for i, header := range headers {
if i >= len(record) {
continue
}
value := strings.TrimSpace(record[i])
if value == "" {
continue
}
// Cari field berdasarkan column map atau nama header
fieldName := s.getFieldName(header)
field, found := t.FieldByName(fieldName)
if !found {
// Coba dengan nama field yang berbeda case
for j := 0; j < t.NumField(); j++ {
f := t.Field(j)
if strings.EqualFold(f.Name, fieldName) {
field = f
found = true
break
}
}
}
if found {
if err := s.setFieldValue(v.FieldByName(field.Name), value); err != nil {
return fmt.Errorf("failed to set field %s: %w", field.Name, err)
}
}
}
return nil
}
// getFieldName mendapatkan nama field dari header CSV
func (s *MasterSeeder) getFieldName(header string) string {
// Cek di column map dulu
if fieldName, exists := s.config.ColumnMap[header]; exists {
return fieldName
}
// Konversi header ke PascalCase
words := strings.Split(header, "_")
for i, word := range words {
words[i] = strings.Title(strings.ToLower(word))
}
return strings.Join(words, "")
}
// setFieldValue mengisi value ke field struct dengan type conversion
func (s *MasterSeeder) setFieldValue(field reflect.Value, value string) error {
if !field.CanSet() {
return nil
}
switch field.Kind() {
case reflect.String:
field.SetString(value)
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
// Handle bigserial (int64)
if value == "" {
field.SetInt(0)
} else {
intVal, err := s.parseInt(value)
if err != nil {
return err
}
field.SetInt(intVal)
}
case reflect.Bool:
boolVal := strings.ToLower(value) == "true" || value == "1"
field.SetBool(boolVal)
case reflect.Interface:
// Untuk field interface{} (seperti Code, Name, Status pada Ethnic)
field.Set(reflect.ValueOf(value))
default:
// Handle pointer types
if field.Kind() == reflect.Ptr {
if value == "" {
field.Set(reflect.Zero(field.Type()))
} else {
// Create new instance of the pointed type
newValue := reflect.New(field.Type().Elem())
if err := s.setFieldValue(newValue.Elem(), value); err != nil {
return err
}
field.Set(newValue)
}
}
}
return nil
}
// parseInt parsing string ke int64
func (s *MasterSeeder) parseInt(value string) (int64, error) {
var intVal int64
_, err := fmt.Sscanf(value, "%d", &intVal)
if err != nil {
return 0, fmt.Errorf("failed to parse int from '%s': %w", value, err)
}
return intVal, nil
}
// setAuditFields mengisi field audit (CreatedAt, UpdatedAt, DeletedAt)
func (s *MasterSeeder) setAuditFields(instance interface{}) {
v := reflect.ValueOf(instance).Elem()
now := time.Now()
// Set CreatedAt jika field ada
if field := v.FieldByName("CreatedAt"); field.IsValid() && field.CanSet() {
if field.Kind() == reflect.Struct {
// Handle time.Time
field.Set(reflect.ValueOf(now))
} else if field.Kind() == reflect.Interface {
field.Set(reflect.ValueOf(now))
}
}
// Set UpdatedAt jika field ada (pointer)
if field := v.FieldByName("UpdatedAt"); field.IsValid() && field.CanSet() {
if field.Kind() == reflect.Ptr {
field.Set(reflect.ValueOf(&now))
}
}
// Set DeletedAt ke nil jika field ada
if field := v.FieldByName("DeletedAt"); field.IsValid() && field.CanSet() {
if field.Kind() == reflect.Ptr {
field.Set(reflect.Zero(field.Type()))
}
}
}
// insertBatch insert batch data ke database
func (s *MasterSeeder) insertBatch(batch []interface{}) error {
return s.db.Transaction(func(tx *gorm.DB) error {
for _, item := range batch {
// Gunakan FirstOrCreate untuk menghindari duplicate
v := reflect.ValueOf(item).Elem()
// Buat kondisi where berdasarkan field utama (Code atau ID)
var condition interface{}
if codeField := v.FieldByName("Code"); codeField.IsValid() {
condition = map[string]interface{}{"Code": codeField.Interface()}
} else if idField := v.FieldByName("Id"); idField.IsValid() {
condition = map[string]interface{}{"Id": idField.Interface()}
}
if condition != nil {
result := tx.Where(condition).FirstOrCreate(item)
if result.Error != nil {
return result.Error
}
} else {
// Jika tidak ada kondisi, insert langsung
if err := tx.Create(item).Error; err != nil {
return err
}
}
}
return nil
})
}
// Helper functions untuk seeding spesifik
// SeedProvinces seed data provinsi dari CSV
func SeedProvinces(db *gorm.DB, csvPath string) error {
config := SeederConfig{
CSVPath: csvPath,
TableName: "Province",
ColumnMap: map[string]string{"Code": "Code", "Name": "Name"},
SkipHeader: true,
BatchSize: 50,
DeleteBefore: true,
}
seeder := NewMasterSeeder(db, config)
// Definisikan struct untuk Province
type Province struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code string `gorm:"uniqueIndex;not null"`
Name string `gorm:"not null"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
return seeder.SeedFromCSV(&Province{})
}
// SeedRegencies seed data kabupaten dari CSV
func SeedRegencies(db *gorm.DB, csvPath string) error {
config := SeederConfig{
CSVPath: csvPath,
TableName: "Regency",
ColumnMap: map[string]string{"Code": "Code", "Province_Code": "ProvinceCode", "Name": "Name"},
SkipHeader: true,
BatchSize: 100,
DeleteBefore: true,
}
seeder := NewMasterSeeder(db, config)
type Regency struct {
Id int64 `gorm:"primaryKey;autoIncrement"`
Code string `gorm:"uniqueIndex;not null"`
ProvinceCode string `gorm:"index;not null"`
Name string `gorm:"not null"`
CreatedAt time.Time
UpdatedAt *time.Time
DeletedAt *time.Time
}
return seeder.SeedFromCSV(&Regency{})
}
// SeedEthnics seed data suku bangsa dari CSV
// func SeedEthnics(db *gorm.DB, csvPath string) error {
// config := SeederConfig{
// CSVPath: csvPath,
// TableName: "Ethnic",
// ColumnMap: map[string]string{"Code": "Code", "Name": "Name", "Status": "Status"},
// SkipHeader: true,
// BatchSize: 50,
// DeleteBefore: true,
// }
// seeder := NewMasterSeeder(db, config)
// // Gunakan struct Ethnic yang sudah ada
// return seeder.SeedFromCSV(&Ethnic{})
// }