Posts

Apache Spark UDF: Over Optimization Issue

Image
What is Apache Spark UDF User-Defined Function (aka UDF) is a feature of Apache Spark SQL to define new Column-based functions that extend the vocabulary of Spark SQL's DSL for transforming Dataset/DataFrame. Defining a UDF: You define a new UDF by defining a function (could be Scala/Python depending upon the underline language) as an input parameter of UDF function. For example: import spark.implicits._ val squared = (s: Long) => {s * s} val squared_udf = spark.udf.register("square", squared) Once it is registered, it can be used as: val df = spark.range(10) df.withColumn("s",squared_udf($"id")).show()

Using Entity Framework 6 Code First with Oracle 11g

Configuration I have used ODP.NET provider for this, which can be easily found on Nuget. The main configuration change here is the EF provider and connection provider. Oracle.ManagedDataAccess.EntityFramework.EFOracleProviderServices is the EF provider and Oracle.ManagedDataAccess.Client.OracleClientFactory is the Oracle connection provider; both providers need to registered. You can registered these either with configuration files, or by code itself. I have created class to registered them, as: public class OracleDbConfiguration : DbConfiguration { public BaseDbConfiguration() { this.SetProviderServices("Oracle.ManagedDataAccess.Client", EFOracleProviderServices.Instance); this.SetProviderFactory("Oracle.ManagedDataAccess.Client", new OracleClientFactory()); } } Now you can use this class with the EF DbCotext class as: [DbConfigurationType(typeof(OracleDbConfiguration))] public class DataContext: DbContext { ... } Oracle Table Consider a simple D

Handling Complex Object with angular-formly

Image
angular-formly angular-formly is an AngularJS module which has a directive to help customize and render JavaScript/JSON configured forms. The  formly-form  directive and the  formlyConfig  service are very powerful and bring unmatched maintainability to your application's forms. For more information visit its  GitHub page . Now when you use formly, it actually provide very basic framework for generating forms. But it provide a great extensibility as well. So, here I am going to use that extensibility. Scenario In many practical scenarios, objects are complex, i.e. they have properties which are not of basic type, but are objects itself. For example consider the following JSON object { "address": { "street": "", "city": "", "state": "" }, "name": "" } The name is of simple type, but address is itself an another object. Now issue is with formly we  can easily g

Dapper.Net One-To-Many mapping

While using Dapper.Net, most common and fundamental scenario is to load the result that form one-to-many relationship. After doing some searching and refactoring the idea multiple times, I get a refactored solution. Scenario Consider a simple customer-order scenario. A customer could have multiple orders, while a order will be associated with only one customer. The models are: public class Customer     {         public String CustomerId { get; set; }         public String CustomerName { get; set; }         public IEnumerable<Order> Orders { get; set; }     } public class Order     {         public string OrderId { get; set; }         public string CustomerId { get; set; }     } The SQL query to fetch both Customer and Order details is, SELECT * FROM Customers c JOIN Orders o ON c.CustomerId = o.CustomerId ORDER BY c.CustomerId Solution Now, the only issue is how to associate each order with its customer. For this we will need a  mapper , to do the mapp